search text file for lines present in another text file and redirect.

rupeshforu3 · 2024-04-09 02:56:51

Hi I am Rupesh from India and I have a system with intel i3 10th gen processor and Asus prime H 510 me motherboard. I have installed Arch Linux and it is working fine.

I have two text files say input1.txt and input2.txt and I want to search input1 file line by line present in another text file called input2.txt and redirect output to another text file missing.txt.

input1 text file consists of lines in the following pattern.

This is a video file 1[abcdef].mp4
This is a video file 2[ghijklm].mp4
This is a video file 3[nopqrst].mp4
This is a video file 4[uvwxyz].mp4

input2 text file consists of lines in the following pattern.

This is a video file 1.mp4
This is a video file 2.mp4
This is a video file 4.mp4

Note that in the second text file one line is missing ie., "This is a video file 3.mp4"

The second text file file consists of lines from first text file last few characters removed.

Now my requirement is I want a script which searches input1.txt text file for lines containing "This is a video file 1" and after that search for "This is a video file 2" and so on.

If a match is found ignore the line. If a match is not found the line must be redirected to another text file called missing.txt.

In the above case missing.txt must contain the following line

This is a video file 3[nopqrst].mp4

You can say that use comm utility or diff utility but they are not applicable in the present situation because some characters are missing in the second text file.

You can even suggest that use text editor like gedit but I have 1000 lines in first text file and 800 lines in second text file.

I want all the 200 remaining missing lines in missing.txt file.

I have a script which has while loop and reads all the lines present in second text file input2.txt.

#!/bin/bash

while read name; do
   
   echo $name

done < input2.txt

But I don't know how to search input1 text file and how to use if construct.

At present I am reading linux operating system and utilities from the beginning and at present I completed how Linux boots and introduction to systemd. It will take time to create a script on my own.

Kindly try to provide a script which searches input1 text file and redirect the unique lines to missing.txt file.

Regards,
Rupesh.

Last edited by rupeshforu3 (2024-04-09 02:57:32)

bulletmark · 2024-04-09 03:43:54

sed 's/\[.*\]//' input1.txt | comm -3 - input2.txt

bulletmark · 2024-04-09 04:00:50

In case you intended the square brackets to mean you want to remove any junk characters after the number and in case you want to ensure the inputs are sorted then use:

sed 's/\([0-9][0-9]*\).*\./\1./' <(sort input1.txt)  | comm -3 - <(sort input2.txt)

rupeshforu3 · 2024-04-09 04:44:23

Thanks for your positive response.

rupeshforu3 · 2024-04-09 09:19:03

Suggestions from other people are as follows

% cat input1.txt | while read LINE; do fgrep "$(echo ${LINE} | sed 's/\[.*//')" input2.txt > /dev/null || echo "${LINE}" >> missing.txt; done

% cat missing.txt
This is a video file 3[nopqrst].mp4

while read -r LINE; do
  grep -F "${LINE%[*}" input2.txt > /dev/null || echo "$LINE" >> missing.txt;
done < input1.txt

wdiff input1.txt input2.txt |grep -v -E '{.*}$'
This is a video file [-3[nopqrst].mp4

while read -r line1; do
    if ! grep -q "$line1" input2.txt; then
        echo "$line1" >> missing.txt
    fi
done < input1.txt

diff -u -d -c 0 input1.txt input2.txt

Trilby · 2024-04-09 12:50:33

You really shouldn't run a loop for this. Bulletmark's approach is best. It may need to be adjusted depending on what is *actually* in input1.txt. Are the square brackets really there, or is that just your way of representing additional characters? If the latter, are these additional characters always just lower case letters and nothing else?

Last edited by Trilby (2024-04-09 13:07:06)

rupeshforu3 · 2024-04-09 14:54:27

Hi the code provided by you is working fine for text files which have lines in short.

The code provided by you and other people are not working on files which have long lines. Long lines mean input1.txt file consists of lines which more than 100 characters.

Bash is not working properly for files which contains more than 600 lines and for files which have long lines which have lines more than 100 columns.

I am going to illustrate what happens actually below.

The script I am using worked properly for short files but it is not working for longer files.

I am going to illustrate step by step what happens actually below.

The following are the contents of the current directory with file size.

[work@Rupesh work3]$ ls -l
total 12
-rw-r--r-- 1 work work 446 Apr  9 19:24 input1.txt
-rw-r--r-- 1 work work 390 Apr  9 19:24 input2.txt
-rwxr-xr-x 1 work work 142 Apr  9 19:10 script.sh

The following are the contents of the text file called input1.txt

[work@Rupesh work3]$ cat input1.txt 
Abhaya Veeranjaneya Swami Temple _ Pallagiri Konda _ Nandigama _ Teerthayatra _ 26th July 2022 _ AP [z-mTkgzd9dE].mp4
Acchamamba Perantalu Temple _ Gudipudi _ Guntur _ Teerthayatra _ 30th March 2021 _AP [BHWdnosP2mw].mp4
Achanteswara Swamy Temple _ Achanta _ West Godavari Dist _ Teerthayatra _ 24th April 2023 _ AP [MgQo5Tm726Y].mp4
Adiyogi Shiva Devalayam _ Coimbatore _ Tamil Nadu _ Teerthayatra _ 18th November 2022 _ETV AP [peKr3q83nvM].mp4

The following are the contents of the text file called input2.txt

[work@Rupesh work3]$ cat input2.txt 
Abhaya Veeranjaneya Swami Temple _ Pallagiri Konda _ Nandigama _ Teerthayatra _ 26th July 2022 _ AP.mp4
Acchamamba Perantalu Temple _ Gudipudi _ Guntur _ Teerthayatra _ 30th March 2021 _AP.mp4
Adiyogi Shiva Devalayam _ Coimbatore _ Tamil Nadu _ Teerthayatra _ 18th November 2022 _ETV AP.mp4

The following are the contents of the script file which previously succeeded but now not.

[work@Rupesh work3]$ cat script.sh 
cat input1.txt | while read LINE; do grep -F "$(echo ${LINE} | sed 's/\[.*//')" input2.txt > /dev/null || echo "${LINE}" >> missing.txt; done

The following shows that the script executed fine

[work@Rupesh work3]$ ./script.sh

The following is the current directory listing including file size.

[work@Rupesh work3]$ ls -l
total 16
-rw-r--r-- 1 work work 446 Apr  9 19:24 input1.txt
-rw-r--r-- 1 work work 390 Apr  9 19:24 input2.txt
-rw-r--r-- 1 work work 446 Apr  9 19:29 missing.txt
-rwxr-xr-x 1 work work 142 Apr  9 19:10 script.sh

After executing the script the missing.txt file must contain only one line which is not found in input2.txt file but instead it consists of all lines present in input1.txt file.

[work@Rupesh work3]$ cat missing.txt 
Abhaya Veeranjaneya Swami Temple _ Pallagiri Konda _ Nandigama _ Teerthayatra _ 26th July 2022 _ AP [z-mTkgzd9dE].mp4
Acchamamba Perantalu Temple _ Gudipudi _ Guntur _ Teerthayatra _ 30th March 2021 _AP [BHWdnosP2mw].mp4
Achanteswara Swamy Temple _ Achanta _ West Godavari Dist _ Teerthayatra _ 24th April 2023 _ AP [MgQo5Tm726Y].mp4
Adiyogi Shiva Devalayam _ Coimbatore _ Tamil Nadu _ Teerthayatra _ 18th November 2022 _ETV AP [peKr3q83nvM].mp4
[work@Rupesh work3]$

Previously I have created a similar thread related to package searching in Arch Linux. The same thing is happening in that case also.

I am requesting all of you to copy the contents of input1.txt and input2.txt and script.sh and run the script in your system.

Even if you get wrong result there is need to modify the original source code of bash software.

This thread must be forwarded to bash shell maintainer so that he can make it work properly.

For any questions I am ready to answer.

seth · 2024-04-09 15:29:54

This has nothing to do with line length or the amount of lines but you omitting blanks in your example.

sed 's/ \[.*\]//' test.1 | comm -3 - test.2

Trilby · 2024-04-09 15:32:18

DO. NOT. LOOP. And this does not need to be forwarded to the maintainers of the bash shell - the shell is fine, you are using it wrong.

And previous solutions "failed" because you did not give us accurate information about what was in the file. Assuming these new examples are real, then the following should be a start:

sed 's/ \[.*\]//' input1.txt | comm -3 - input2.txt

That gives the one name from input2.txt that isn't also included in input1.txt. It lacks the characters in the square brackets (is this a hash?), but it'd be trivial to add another step to add that back if needed, e.g.:

sed 's/.*/&\t&/;s/ \[.*\]//' input1.txt | join -j1 -v1 -t\t - input2.txt | cut -f2

(edit: this also assumes the inputs are sorted)

Last edited by Trilby (2024-04-09 15:44:11)

rupeshforu3 · 2024-04-09 16:13:34

hi i am providing the output of the above two commands below

[work@Rupesh work2]$ sed 's/ \[.*\]//' input1.txt | comm -3 - input2.txt
Sadguru Sai Tatva Gyana Mandiram _ Seethanagaram _ W.G._ Teerthayatra _ 30th May 2019_AP.mp4
comm: file 1 is not in sorted order
comm: file 2 is not in sorted order
Sri Agastheeswarar Swamy Temple _ Proddatur _ Teerthayatra _ 5th November 2019 _ Full Episode.mp4
Sri Agastheswara Swamy Temple _ Proddatur _ Kadapa_ _ 4th June 2018 _ ETV AP.mp4
	Sri Andhrashirdi Saibaba Temple _Balabhadrapuram_W.G.Dist._Teerthayatra _ 12th  November 2020_ETV AP.mp4
Sri Andhra Shirdi Saibaba Temple _ Teerthayatra _ 4th October 2018 _ Full Episode _ ETV AP.mp4
Sri Andhrashirdi Saibaba Temple _Balabhadrapuram_W.G.Dist._Teerthayatra _ 12th  November 2020_ETV AP.mp4
Sri Anjaneya Swamy Temple _ Machilipatnam _ Krishna District _ Teerthayatra _ 26th June 2018 _ Full.mp4
Sri AnjaneyaSwamy Temple _ Muradi _ Anantapur _ Teerthayatra _ 20th August 2019_ ETV AP.mp4
Sri Anjaneyaswamy Temple _Nemakallu _Anantapur _Teerthayatra _ 10th December 2019_ ETV AP.mp4
Sri Banashankari Temple _ Banashankari _ Karnataka _ Teerthayatra _ 28th September 2018 _AP.mp4
Sri Bhusametha Vijaya Venkateswara Swamy Temple _Vizag _ Teerthayatra _23rd February 2019 _AP.mp4
Sri Chengalamma Parameswari Temple _ Teerthayatra _ 1st June 2018 _ Full Episode _ ETV AP.mp4
Sri Chinatala Venkateswara Swami Temple _ Anamtapuram District _ Teerthayatra _  1st August 2020.mp4
	Sri Chintala Venkataramana Swamy Temple_Tadipatri _Anantapur District _Teerthayatra_13th August 2022.mp4
Sri Chintala Venkataramana Swamy Temple _ Tadipatri _ Teerthayatra _ 5th May 2018 _ Full Episode.mp4
Sri Chintala Venkataramana Swamy Temple_Tadipatri _Anantapur District _Teerthayatra_13th August 2022.mp4
Sri Chintamani Ganpati Temple _ Visakha District _ Teerthayatra _ 9th September 2019 _ Full Episode.mp4
Sri Dattatreya Swami Temple _ Eluru _ Teerthayatra _ 22nd December 2018 _ Full Episode _ ETV AP.mp4
Sri Duuddhaarii Sriiraamapamchaayatana Temple _ Teerthayatra _ 10th May 2020 _ Full Epi.mp4
Sri Gangamma Temple _ Boyakonda _ Chittoor _ Teerthayatra _ 19th October 2018 _ ETV AP.mp4
Sri Gavi Yoga Narasimha Swamy Devasthanam _ Kaivara _ Karnataka _ Teerthayatra _ 23rd December 2018.mp4
	Sri Gayatridevi Temple _ Telaprolu _ Krishna Dist._ Teerthayatra _ 19th October 2020 _ ETV AP.mp4
Sri Gayatri Shakti Peetam _ Nagpur _ Maharashtra _ Teerthayatra _ 12th October 2018 _ ETV AP.mp4
Sri Gayatridevi Temple _ Krushna Dist. _ Teerthayatra _ 2nd October 2019 _ ETV AP.mp4
Sri Gayatridevi Temple _ Telaprolu _ Krishna Dist._ Teerthayatra _ 19th October 2020 _ ETV AP.mp4
Sri Hanuman Temple _ Jabili Theertham _ Chittoor District _ Teerthayatra _ 12th June 2018.mp4
Sri Jagannatha Swamy Temple _Vadali _ Krishna dist _Teerthayatra _ 4th July 2018 _ ETV AP.mp4
Sri Kalabhairava Swami Temple _ Ujjaini _ Madhya Pradesh _ Teerthayatra _ 5th December 2023 _ ETV AP.mp4
Sri Kalabhairaveshwara Swamy Temple _ Bengaluru _ Karnataka _ Teerthayatra _ 11th December 2021.mp4
	Sri Kalimata Temple _ Visakhapatnam _ RK Beach _ Teerthayatra _ 12th June 2020 _ Full Episode.mp4
	Sri Kalimata Temple _ Visakhapatnam _ RK Beach _ Teerthayatra _ 16th June 2023 _ AP.mp4
Sri Kali Mata Temple _ Vishakapatnam _ Teerthayatra _ 4th January 2019 _ Full Episode _ ETV AP.mp4
Sri Kalimata Temple _ Visakhapatnam _ RK Beach _ Teerthayatra _ 12th June 2020 _ Full Episode.mp4
Sri Kalimata Temple _ Visakhapatnam _ RK Beach _ Teerthayatra _ 16th June 2023 _ AP.mp4
Sri Kalimatha Temple _ Visakhapatnam _ R.K.Beach _ Teerthayatra _ 17th June 2022 _AP.mp4
Sri Kalyana KodandaramaSwami Temple _ Turpu Godavari District _  Teerthayatra _ 4th November 2020.mp4
Sri Kalyana Kothandaramaswamy Temple_Turpu Godavari Dist_Teerthayatra _9th October 2019_Full Epi.mp4
Sri Kanaka Durga Devi Temple _Pallagiri Konda _Nandigama _ Teerthayatra _ 27th April 2018_ AP.mp4
Sri Kanaka Durga Malleswara Swamy Temple _ Tadinaada _ Teerthayatra _ 10th November 2023  _ ETV AP.mp4
	Sri Kanakadurga MalleswaraSwamy Temple _ Tadinada _ Krishna _Teerthayatra _ 12th April 2019_ ETV AP.mp4
	Sri Kanakasurabheaswara Swami Temple _ Prakasam District _ Teerthayatra _ 27th November 2021 _  ETV.mp4
Sri Kanaka Surabheswara Swamy Temple _ Surabheswara Kona _Prakasam _Teerthayatra _ 16th July 2018.mp4
Sri Kanakadurga MalleswaraSwamy Temple _ Tadinada _ Krishna _Teerthayatra _ 12th April 2019_ ETV AP.mp4
Sri Kanakadurgadevi Temple _ Indrakeeladri _ Vijayawada _ Teerthayatra _ 26th September 2022 _ETV AP.mp4
Sri Kanakasurabheaswara Swami Temple _ Prakasam District _ Teerthayatra _ 27th November 2021 _  ETV.mp4
Sri Kashi Vishwanathswamy Temple _ Kalavara _ Karnataka _ Teerthayatra _ 5th November 2023 _ ETV AP.mp4
Sri Kasi Viswanatha Swamy Temple _ Kalavara _ Karnataka _ Teerthayatra _ 18th October 2021 _ ETV AP.mp4
Sri Kasi Viswanatha Swamy Temple _ Kalavara _ Karnataka _ Teerthayatra _ 21st October 2018_ ETV AP.mp4
Sri Kasi Viswanatha Temple _ Kashi _ Uttar Pradesh _ Teerthayatra _ 11th November 2021 _ ETV  AP.mp4
Sri Kasi Viswanathar Temple _ Varanasi _Uttar Pradesh _ Teerthayatra _ 18th February 2023 _ ETV _ AP.mp4
Sri Kasivisweswara Swami Temple _ Ponugupadu _ Guntur _ Teerthayatra _ 1st March 2021 _ ETV AP.mp4
Sri Kodanda Rama Swamy Temple _ Moturu _ Krishna District _ Teerthayatra _ 1st March 2023 _ ETV AP.mp4
	Sri Kodanda Ramaswamy Temple _Buchireddipalem _Nellore _ Teerthayatra _ 29th June 2022 _AP.mp4
	Sri Kodandarama Swami Ramalingeshwara Swami Temple _ Teerthayatra _ 7th June 2023 _ ETV AP.mp4
Sri Kodanda Ramaswamy Temple _ Buchireddypalem _ Nellore _ Teerthayatra _ 1st August 2018 _ ETV AP.mp4
Sri Kodanda Ramaswamy Temple _Buchireddipalem _Nellore _ Teerthayatra _ 29th June 2022 _AP.mp4
Sri Kodandarama Swami Ramalingeshwara Swami Temple _ Teerthayatra _ 7th June 2023 _ ETV AP.mp4
	Sri Kodandarama-RamalingeswaraSwami Devasthanam_ Krishna Dist._ Teerthayatra_ 22nd June 2022 _ETV AP.mp4
	Sri Kodandaramaswamy Temple _ Sriramatheertham _ Vizianagaram _ Teerthayatra _ 17th July 2018 _ AP.mp4
	Sri Kodandaramaswamy Temple _Sriramatheertham _Vizianagaram _Teerthayatra _ 5th June 2019 _ETV AP.mp4
Sri Kodandarama Swamy Temple _ Vontimitta _ Kadapa _ Teerthayatra _ 26th April 2021 _ ETV AP.mp4
Sri Kodandarama Swamy Temple_Nellore District_Teerthayatra_16th January 2019_ Full Episode _ ETV AP.mp4
Sri Kodandarama-RamalingeswaraSwami Devasthanam_ Krishna Dist._ Teerthayatra_ 22nd June 2022 _ETV AP.mp4
Sri Kodandaramaswamy Temple _ Sriramatheertham _ Vizianagaram _ Teerthayatra _ 17th July 2018 _ AP.mp4
Sri Kodandaramaswamy Temple _Sriramatheertham _Vizianagaram _Teerthayatra _ 5th June 2019 _ETV AP.mp4
Sri Krishna Balarama Mandiram Iskcon_ Hubli _ Karnataka _ Teerthayatra _ 29th August 2021 _ ETV AP.mp4
Sri Kurmanatha Swamy Temple _ Srikurmam _ Srikakulam District _ Teerthayatra _ 25th June 2022 _ ETV.mp4
Sri Lakshmaneswara Swamy Temple _ Lakshmaneswaram _ W.G._ Teerthayatra _ 4th February 2019 _ AP.mp4
Sri Lakshmi Ganapathi Temple _Bikkavolu _ Turpu Godavari District _ Teerthayatra _ 8th November 2023.mp4
Sri Lakshmi Narasimha Swamy Devasthanam _ Yadadri Mahakshetram _ Teerthayatra _ 28th February 2023.mp4
Sri Lakshmi Narasimha Swamy Temple _ Amtarveadi _ Teerthayatra _ 23rd February 2021 _ Full Episode.mp4
Sri Lakshmi Narasimha Swamy Temple _ Antarvedi _ Teerthayatra _ 1st February 2023 _ ETV P.mp4
Sri Lakshmi Narasimha Swamy Temple _ Nallagonda _ Nellore _Teerthayatra _ 27th October 2020 _AP.mp4
Sri Lakshmi Narasimha Swamy Temple _ Nellore District _ Teerthayatra _ 30th August 2023 _ ETV AP.mp4
Sri Lakshmi Narasimha Swamy Temple _ Turpu Godavari District _ Teerthayatra _ 21st November 2023 .mp4
Sri Lakshmi Narasimha Swamy Temple _Antarvedi _Konaseema District _ Teerthayatra _ 10th October 2023.mp4
Sri Lakshmi Narasimha Swamy Temple, Koppal, Karnataka, _ 25th June 2023 _ Full Episode _ ETV AP.mp4
Sri Lakshmi Narasimhaswami Temple _ Yadadri Bhuvanagiri District _ Teerthayatra _ 23rd March 2021.mp4
Sri Lakshmi NarasimhaSwamy Temple _Antervedi _Teerthayatra_ 13th December 2022 _Full Episode _ETV AP.mp4
Sri Madana Gopala Swamy Temple _ Teerthayatra _ 11th July 2018 _ Full Episode _ ETV Andhra Pradesh.mp4
Sri Maddaneswarswamy Temple _ 74,Udegolam _ Anantapuram _ Teerthayatra _ 13th November 2018_ ETV AP.mp4
Sri Maddi Anjaneya Swamy Temple_Teerthayatra_ 11th September 2018 _Full Episode_ ETV Andhra Pradesh.mp4
Sri Maddi Anjaneya SwamyTemple _ Guravaigudem _ Teerthayatra _ 17th January 2023 _ Full Episode _ETV.mp4
Sri Madhaveswari Temple _ Prayaga _ Utarakhand _ Teerthayatra _ 12th February 2021 _ Full Episode.mp4
Sri Mahaganapathi Temple _Brahmapur _Odisha _ Teerthayatra_ 13th September 2021 _Full Episode_ETV AP.mp4
Sri Mahalakshmi Devi Mandiram _ Thane _ Maharashtra _ Teerthayatra _ 1st September 2023 _AP.mp4
Sri Mahalakshmi Temple _ Kolhapur _ Maharashtra _ Teerthayatra _ 18th August 2023 _ ETV AP.mp4
Sri Mahalakshmi Temple _ Kolhapur _ Maharashtra _ Teerthayatra _ 24th April 2020_ETV AP.mp4
Sri Mangaladevi Temple _ Mangalore _ Karnataka _ Teerthayatra _ 22nd August 2023 _ AP.mp4
Sri Marakata Mahalakshmi Temple _ Kadiri _Anantapur _ Teerthayatra _ 17th July 2020.mp4
Sri Marakatha Mahalakshmi Temple _ Kadiri _ Anantapur Dist _ 16th October 2020 _AP.mp4
Sri Marakatha Mahalakshmi Temple _ Teerthayatra _ 13th July 2018 _ Full Episode _ ETV AP.mp4
Sri Markandeshwara Swamy Temple _ Gottipadu _ Guntur _ Teerthayatra _ 19th September 2022 _ ETV AP.mp4
Sri Matha Dhari Devi Temple _ Teerthayatra _ 21st September 2018 _ Full Episode _ ETV Andhra Pradesh.mp4
Sri Mavullamma Temple _ Bhimavaram _ W.G._ Teerthayatra _ 16th September 2022 _ AP.mp4
Sri Mulasthaneswara Swamy Temple _ Nellore _ Teerthayatra _ 14th May 2018 _ Full Episode.mp4
comm: input is not in sorted order
[work@Rupesh work2]$

The head and tail of the above output is as below

[work@Rupesh work2]$ head input3.txt 
[work@Rupesh work2]$ sed 's/ \[.*\]//' input1.txt | comm -3 - input2.txt
Sadguru Sai Tatva Gyana Mandiram _ Seethanagaram _ W.G._ Teerthayatra _ 30th May 2019_AP.mp4
comm: file 1 is not in sorted order
comm: file 2 is not in sorted order
Sri Agastheeswarar Swamy Temple _ Proddatur _ Teerthayatra _ 5th November 2019 _ Full Episode.mp4
Sri Agastheswara Swamy Temple _ Proddatur _ Kadapa_ _ 4th June 2018 _ ETV AP.mp4
	Sri Andhrashirdi Saibaba Temple _Balabhadrapuram_W.G.Dist._Teerthayatra _ 12th  November 2020_ETV AP.mp4
Sri Andhra Shirdi Saibaba Temple _ Teerthayatra _ 4th October 2018 _ Full Episode _ ETV AP.mp4
Sri Andhrashirdi Saibaba Temple _Balabhadrapuram_W.G.Dist._Teerthayatra _ 12th  November 2020_ETV AP.mp4
Sri Anjaneya Swamy Temple _ Machilipatnam _ Krishna District _ Teerthayatra _ 26th June 2018 _ Full.mp4
[work@Rupesh work2]$ tail input3.txt 
Sri Marakata Mahalakshmi Temple _ Kadiri _Anantapur _ Teerthayatra _ 17th July 2020.mp4
Sri Marakatha Mahalakshmi Temple _ Kadiri _ Anantapur Dist _ 16th October 2020 _AP.mp4
Sri Marakatha Mahalakshmi Temple _ Teerthayatra _ 13th July 2018 _ Full Episode _ ETV AP.mp4
Sri Markandeshwara Swamy Temple _ Gottipadu _ Guntur _ Teerthayatra _ 19th September 2022 _ ETV AP.mp4
Sri Matha Dhari Devi Temple _ Teerthayatra _ 21st September 2018 _ Full Episode _ ETV Andhra Pradesh.mp4
Sri Mavullamma Temple _ Bhimavaram _ W.G._ Teerthayatra _ 16th September 2022 _ AP.mp4
Sri Mulasthaneswara Swamy Temple _ Nellore _ Teerthayatra _ 14th May 2018 _ Full Episode.mp4
comm: input is not in sorted order
[work@Rupesh work2]$

[work@Rupesh work2]$ sed 's/.*/&\t&/;s/ \[.*\]//' input1.txt | join -j1 -v1 -t\t - input2.txt | cut -f2
join: Sadguru Sai Tatva Gyana Mandiram _ Seethanagaram _ W.G._ Teerthayatra _ 30th May 2019_AP.mp4
-:245: is not sorted: Sai Baba Mandiram _ Bhimavaram _ Teerthayatra _ 5th October 2023 _ Andhra Pradesh.mp4
join: input2.txt:244: is not sorted: Sai Baba Mandiram _ Bhimavaram _ Teerthayatra _ 5th October 2023 _ Andhra Pradesh.mp4
Sri Andhra Shirdi Saibaba Temple _ Teerthayatra _ 4th October 2018 _ Full Episode _ ETV AP.mp4
Sri Andhrashirdi Saibaba Temple _Balabhadrapuram_W.G.Dist._Teerthayatra _ 12th  November 2020_ETV AP.mp4
Sri Anjaneya Swamy Temple _ Machilipatnam _ Krishna District _ Teerthayatra _ 26th June 2018 _ Full.mp4
Sri AnjaneyaSwamy Temple _ Muradi _ Anantapur _ Teerthayatra _ 20th August 2019_ ETV AP.mp4
Sri Anjaneyaswamy Temple _Nemakallu _Anantapur _Teerthayatra _ 10th December 2019_ ETV AP.mp4
Sri Banashankari Temple _ Banashankari _ Karnataka _ Teerthayatra _ 28th September 2018 _AP.mp4
Sri Bhusametha Vijaya Venkateswara Swamy Temple _Vizag _ Teerthayatra _23rd February 2019 _AP.mp4
Sri Chengalamma Parameswari Temple _ Teerthayatra _ 1st June 2018 _ Full Episode _ ETV AP.mp4
Sri Chinatala Venkateswara Swami Temple _ Anamtapuram District _ Teerthayatra _  1st August 2020.mp4
Sri Duuddhaarii Sriiraamapamchaayatana Temple _ Teerthayatra _ 10th May 2020 _ Full Epi.mp4
Sri Gangamma Temple _ Boyakonda _ Chittoor _ Teerthayatra _ 19th October 2018 _ ETV AP.mp4
Sri Gavi Yoga Narasimha Swamy Devasthanam _ Kaivara _ Karnataka _ Teerthayatra _ 23rd December 2018.mp4
Sri Hanuman Temple _ Jabili Theertham _ Chittoor District _ Teerthayatra _ 12th June 2018.mp4
Sri Kalabhairava Swami Temple _ Ujjaini _ Madhya Pradesh _ Teerthayatra _ 5th December 2023 _ ETV AP.mp4
Sri Kalabhairaveshwara Swamy Temple _ Bengaluru _ Karnataka _ Teerthayatra _ 11th December 2021.mp4
Sri Kali Mata Temple _ Vishakapatnam _ Teerthayatra _ 4th January 2019 _ Full Episode _ ETV AP.mp4
Sri Kalimata Temple _ Visakhapatnam _ RK Beach _ Teerthayatra _ 12th June 2020 _ Full Episode.mp4
Sri Kalimata Temple _ Visakhapatnam _ RK Beach _ Teerthayatra _ 16th June 2023 _ AP.mp4
Sri Kalimatha Temple _ Visakhapatnam _ R.K.Beach _ Teerthayatra _ 17th June 2022 _AP.mp4
Sri Kalyana KodandaramaSwami Temple _ Turpu Godavari District _  Teerthayatra _ 4th November 2020.mp4
Sri Kalyana Kothandaramaswamy Temple_Turpu Godavari Dist_Teerthayatra _9th October 2019_Full Epi.mp4
Sri Kanaka Durga Devi Temple _Pallagiri Konda _Nandigama _ Teerthayatra _ 27th April 2018_ AP.mp4
Sri Kanaka Durga Malleswara Swamy Temple _ Tadinaada _ Teerthayatra _ 10th November 2023  _ ETV AP.mp4
Sri Kanaka Surabheswara Swamy Temple _ Surabheswara Kona _Prakasam _Teerthayatra _ 16th July 2018.mp4
Sri Kanakadurga MalleswaraSwamy Temple _ Tadinada _ Krishna _Teerthayatra _ 12th April 2019_ ETV AP.mp4
Sri Kanakadurgadevi Temple _ Indrakeeladri _ Vijayawada _ Teerthayatra _ 26th September 2022 _ETV AP.mp4
Sri Kanakasurabheaswara Swami Temple _ Prakasam District _ Teerthayatra _ 27th November 2021 _  ETV.mp4
Sri Kasi Viswanatha Swamy Temple _ Kalavara _ Karnataka _ Teerthayatra _ 18th October 2021 _ ETV AP.mp4
Sri Kasi Viswanatha Swamy Temple _ Kalavara _ Karnataka _ Teerthayatra _ 21st October 2018_ ETV AP.mp4
Sri Kasi Viswanatha Temple _ Kashi _ Uttar Pradesh _ Teerthayatra _ 11th November 2021 _ ETV  AP.mp4
Sri Kasi Viswanathar Temple _ Varanasi _Uttar Pradesh _ Teerthayatra _ 18th February 2023 _ ETV _ AP.mp4
Sri Kasivisweswara Swami Temple _ Ponugupadu _ Guntur _ Teerthayatra _ 1st March 2021 _ ETV AP.mp4
Sri Kodanda Rama Swamy Temple _ Moturu _ Krishna District _ Teerthayatra _ 1st March 2023 _ ETV AP.mp4
Sri Kodanda Ramaswamy Temple _ Buchireddypalem _ Nellore _ Teerthayatra _ 1st August 2018 _ ETV AP.mp4
Sri Kodanda Ramaswamy Temple _Buchireddipalem _Nellore _ Teerthayatra _ 29th June 2022 _AP.mp4
Sri Kodandarama Swami Ramalingeshwara Swami Temple _ Teerthayatra _ 7th June 2023 _ ETV AP.mp4
Sri Kodandarama Swamy Temple _ Buchireddypalem _ Nellore _ Teerthayatra _ 20th January 2021 _ AP.mp4
Sri Kodandarama Swamy Temple _ Buchireddypalem _ Nellore_ Teerthayatra _ 16th September 2020 _AP.mp4
Sri Kodandarama Swamy Temple _ Vontimitta _ Kadapa _ Teerthayatra _ 26th April 2021 _ ETV AP.mp4
Sri Kodandarama Swamy Temple_Nellore District_Teerthayatra_16th January 2019_ Full Episode _ ETV AP.mp4
Sri Kodandarama-RamalingeswaraSwami Devasthanam_ Krishna Dist._ Teerthayatra_ 22nd June 2022 _ETV AP.mp4
join: input is not in sorted orderSri Kodandaramaswamy Temple _ Sriramatheertham _ Vizianagaram _ Teerthayatra _ 17th July 2018 _ AP.mp4

Sri Kodandaramaswamy Temple _Sriramatheertham _Vizianagaram _Teerthayatra _ 5th June 2019 _ETV AP.mp4
Sri Krishna Balarama Mandiram Iskcon_ Hubli _ Karnataka _ Teerthayatra _ 29th August 2021 _ ETV AP.mp4
Sri Kurmanatha Swamy Temple _ Srikurmam _ Srikakulam District _ Teerthayatra _ 25th June 2022 _ ETV.mp4
Sri Lakshmaneswara Swamy Temple _ Lakshmaneswaram _ W.G._ Teerthayatra _ 4th February 2019 _ AP.mp4
Sri Lakshmi Narasimha Swamy Devasthanam _ Yadadri Mahakshetram _ Teerthayatra _ 28th February 2023.mp4
Sri Lakshmi Narasimha Swamy Temple _ Amtarveadi _ Teerthayatra _ 23rd February 2021 _ Full Episode.mp4
Sri Lakshmi Narasimha Swamy Temple _ Nallagonda _ Nellore _Teerthayatra _ 27th October 2020 _AP.mp4
Sri Lakshmi Narasimha Swamy Temple _ Turpu Godavari District _ Teerthayatra _ 21st November 2023 .mp4
Sri Lakshmi Narasimha Swamy Temple _Antarvedi _Konaseema District _ Teerthayatra _ 10th October 2023.mp4
Sri Lakshmi Narasimha Swamy Temple, Koppal, Karnataka, _ 25th June 2023 _ Full Episode _ ETV AP.mp4
Sri Lakshmi Narasimhaswami Temple _ Yadadri Bhuvanagiri District _ Teerthayatra _ 23rd March 2021.mp4
Sri Lakshmi NarasimhaSwamy Temple _Antervedi _Teerthayatra_ 13th December 2022 _Full Episode _ETV AP.mp4
Sri Maddaneswarswamy Temple _ 74,Udegolam _ Anantapuram _ Teerthayatra _ 13th November 2018_ ETV AP.mp4
Sri Maddi Anjaneya Swamy Temple_Teerthayatra_ 11th September 2018 _Full Episode_ ETV Andhra Pradesh.mp4
Sri Maddi Anjaneya SwamyTemple _ Guravaigudem _ Teerthayatra _ 17th January 2023 _ Full Episode _ETV.mp4
Sri Madhaveswari Temple _ Prayaga _ Utarakhand _ Teerthayatra _ 12th February 2021 _ Full Episode.mp4
Sri Mahalakshmi Devi Mandiram _ Thane _ Maharashtra _ Teerthayatra _ 1st September 2023 _AP.mp4
Sri Mahalakshmi Temple _ Kolhapur _ Maharashtra _ Teerthayatra _ 18th August 2023 _ ETV AP.mp4
Sri Mahalakshmi Temple _ Kolhapur _ Maharashtra _ Teerthayatra _ 24th April 2020_ETV AP.mp4
Sri Mangaladevi Temple _ Mangalore _ Karnataka _ Teerthayatra _ 22nd August 2023 _ AP.mp4
Sri Markandeshwara Swamy Temple _ Gottipadu _ Guntur _ Teerthayatra _ 19th September 2022 _ ETV AP.mp4
[work@Rupesh work2]$

The head and tail of the above output is as below

[work@Rupesh work2]$ head input3.txt 
[work@Rupesh work2]$ sed 's/.*/&\t&/;s/ \[.*\]//' input1.txt | join -j1 -v1 -t\t - input2.txt | cut -f2
join: Sadguru Sai Tatva Gyana Mandiram _ Seethanagaram _ W.G._ Teerthayatra _ 30th May 2019_AP.mp4
-:245: is not sorted: Sai Baba Mandiram _ Bhimavaram _ Teerthayatra _ 5th October 2023 _ Andhra Pradesh.mp4
join: input2.txt:244: is not sorted: Sai Baba Mandiram _ Bhimavaram _ Teerthayatra _ 5th October 2023 _ Andhra Pradesh.mp4
Sri Andhra Shirdi Saibaba Temple _ Teerthayatra _ 4th October 2018 _ Full Episode _ ETV AP.mp4
Sri Andhrashirdi Saibaba Temple _Balabhadrapuram_W.G.Dist._Teerthayatra _ 12th  November 2020_ETV AP.mp4
Sri Anjaneya Swamy Temple _ Machilipatnam _ Krishna District _ Teerthayatra _ 26th June 2018 _ Full.mp4
Sri AnjaneyaSwamy Temple _ Muradi _ Anantapur _ Teerthayatra _ 20th August 2019_ ETV AP.mp4
Sri Anjaneyaswamy Temple _Nemakallu _Anantapur _Teerthayatra _ 10th December 2019_ ETV AP.mp4
Sri Banashankari Temple _ Banashankari _ Karnataka _ Teerthayatra _ 28th September 2018 _AP.mp4
[work@Rupesh work2]$ tail input3.txt 
Sri Madhaveswari Temple _ Prayaga _ Utarakhand _ Teerthayatra _ 12th February 2021 _ Full Episode.mp4
Sri Mahalakshmi Devi Mandiram _ Thane _ Maharashtra _ Teerthayatra _ 1st September 2023 _AP.mp4
Sri Mahalakshmi Temple _ Kolhapur _ Maharashtra _ Teerthayatra _ 18th August 2023 _ ETV AP.mp4
Sri Mahalakshmi Temple _ Kolhapur _ Maharashtra _ Teerthayatra _ 24th April 2020_ETV AP.mp4
Sri Mangaladevi Temple _ Mangalore _ Karnataka _ Teerthayatra _ 22nd August 2023 _ AP.mp4
Sri Markandeshwara Swamy Temple _ Gottipadu _ Guntur _ Teerthayatra _ 19th September 2022 _ ETV AP.mp4
[work@Rupesh work2]$

however you have not recognized what I want ie., lines unique in input1.txt including square brackets and its contents eg "This is a video file 3[nopqrst].mp4"

thanks for your patience.

seth · 2024-04-09 16:36:39

I've think I've recognized that you want us to do your homework…

Chop away the suffix w/ sed and use that as pattern file for an inverse grep.
Come back with an approach that shows you understand the idea and that is not "suggestions from other people"

Trilby · 2024-04-09 18:07:25

Yes, we understand perfectly, you do not:

Trilby wrote:

(edit: this also assumes the inputs are sorted)

Note that your manner of posting can be seen as somewhat rude. I gather English is not your first language and perhaps the tone is the result of a language barrier and not intended. But as seth touched on above, you do not seem to be putting in your share of effort to solving your problem; that is not the result of a language barrier. We are here to help, but do not expect us to do all your thinking for you.

Last edited by Trilby (2024-04-09 18:12:10)

rupeshforu3 · 2024-04-10 01:32:18

Hi I think if we follow the correct procedure the issue can be resolved and I think that bash is working fine.

1) first for each line in input1.txt the square brackets and the characters with in square brackets must be removed.

2) The trailing space character must be removed.

3) The resultant line must be searched with in input2.txt.

4) If the line is found in input2.txt the line must be ignored.

5) If the line is not found the original line including square brackets and it's contents from input1.txt must be redirected to missing.txt file.

Previously I have used the following code to remove square brackets including its contents from file name with the following code.

for x in *.mp4; do mv "$x" "${x// \[*\]/}"; done

We must modify the above code and use it in step 1 and step 2.

I have thought how to solve the problem and found the above. I am sharing this to you to share my effort.

Let me try if I can succeed. Here the problem is I am not as talented person like you. I have not studied sed awk grep etc.,.

From the past 4 days I am working on the same problem and not succeeded and so I have made comments that bash is not working properly. Sorry for that.

Last edited by rupeshforu3 (2024-04-10 03:24:37)

seth · 2024-04-10 07:29:29

You're just re-stating the task and this isn't about talent

You wrote:

you have not recognized what I want ie., lines unique in input1.txt including square brackets and its contents eg "This is a video file 3[nopqrst].mp4"

But Trilby explicitly addressed that in #9
He also pointed out a necessary precondition of the actual data for comm to work and comm told you that this isn't met, but you've ignored that, too.

So I explained an alternative approach that doesn't require this precondition and will print the missing files from input1.txt directly - whether that's more efficient than the comm approach will depend on the actual data set, but it's just 51 chars (47 if you remove unnecessary blanks) and likely easier to understand/implement.
You've ignored that, too.

We must modify the above code and use it in step 1 and step 2.

No. Get that out of your head.

Last edited by seth (2024-04-10 07:29:41)

Trilby · 2024-04-10 12:29:46

This thread is a monologue. Anyone other than the OP are mere spectators. We can comment, but our comments have no apparent impact on the trajectory of the monologue. This should really be in the OP's blog if they just want a record of their own efforts.

rupeshforu3 · 2024-04-10 13:45:45

Hi finally I have created a small script based upon my past experience and suggestions from the people like you and it is as follows.

#!/bin/bash

rm missing.txt

while read LINE; do
   
#   echo $LINE
   NAME="${LINE// \[*\]/}"
   NAME2=${NAME%????}
   NAME3="$(echo -e "${NAME2}" | sed -e 's/^[[:space:]]*//')"
   NAME4="${NAME3}.mp4"
#   echo $NAME2
#   echo ${NAME}
   grep -F "$(echo ${NAME4})" input2.txt > /dev/null || echo "${LINE}" >> missing.txt
done < input1.txt

Many of you suggested single line code which works at some times and remaining not.

But the above code works in all situations. I have created this script step by step.

1) First I have removed square brackets including its contents from each line of input1.txt using the following code.

NAME="${LINE// \[*\]/}"

2) I tried to remove trailing space from each line using the following code.

NAME2=${NAME%????}
NAME3="$(echo -e "${NAME2}" | sed -e 's/^[[:space:]]*//')"
NAME4="${NAME3}.mp4"

3) Finally I have searched all lines of input2.txt containing the contents of variable NAME4 using the following code.

grep -F "$(echo ${NAME4})" input2.txt > /dev/null || echo "${LINE}" >> missing.txt

The most important step is to remove square brackets and including its contents through step 1.

The above script is working fine except step 2 I mean the code provided in step 2 can't remove trailing space.

Trailing space means.

input1.txt consists of some lines with pattern

This is a video file 1 [abcdefg].mp4

input2.txt consists of lines with pattern

This is video file 1.mp4

Generally the above script must ignore this line and it must not redirect this particular line to missing.txt file but it is being redirected.

I have included code to remove trailing white space before performing grep. I think that trailing white space has been removed but still redundant line is being redirected to missing.txt.

I think that if the above script run properly the missing text file must contain only 80 lines but at present it consists of 165 lines.

Something is better than nothing and that too you may be irritated if I go on saying not working not working etc.,. So I am aborting the current work.

If you still want to know what's not working then my answer is search is not performed well for lines containing trailing white space before square brackets in input1.txt.

Thanks for your patience.

seth · 2024-04-10 13:52:16

Trilby's going into cardiac arrest in 3…2…

sed 's/\.mp4$//g' input2.txt | grep -vFf - input1.txt

I think that if the above script run properly the missing text file must contain only 80 lines but at present it consists of 165 lines.

No.

Last edited by seth (2024-04-10 14:07:41)

rupeshforu3 · 2024-04-10 14:51:41

I am in the learning stage and so I may get doubts please try to clarify with patience.

As I am still in the learning stage I don't how to use utilities exactly.

I agree that everything works fine under Arch Linux but I don't know how to use them.

Trilby · 2024-04-10 16:51:13

rupeshforu3 wrote:

Hi finally I have created a small script based upon my past experience ~~and suggestions from the people like you~~ and it is as follows.

FTFY.

seth · 2024-04-10 19:05:20

I am in the learning stage

Good.
Have you tried the command in #17 on your data?
Do you understand what it does?

Do you understand what's wrong with the files resulting in the failure of the comm approach (hint: comm actually told you)?

solskog · 2024-04-12 08:35:37

how about python? The idea is to create two dictionary then look it up with the keys.

#!/bin/env python
import sys
class Missing:
    def __init__(self,*argv):
        self.argv = argv
        self.args = argv[0]
        self.argc = len(self.args)
    def missing(self):
        if self.argc < 3:
            print('missing file1 file2', out=sys.stderr)
            sys.exit(1)
        file1 = self.args[1]
        file2 = self.args[2]
        content1 = []
        table1 = {}
        content2 = []
        table2 = {}
        table3 = {}
        with open(file1,'r') as fh:
            content1 = fh.read().rstrip('\n').split('\n')
        with open(file2,'r') as fh:
            content2 = fh.read().rstrip('\n').split('\n')
        for line in content1:
            record = line.split('[')
            value = ' [' + record.pop()
            key = ''.join(record).rstrip(' ')
            table1.update({key:value})
        for line in content2:
            record = line.split('.')
            value = '.' + record.pop()
            key = ''.join(record)
            table2.update({key:value})
        for key,value in table1.items():
            if key in table2:continue
            print(key + value)
if __name__ == '__main__':
    self = Missing(sys.argv)
    self.missing()

Last edited by solskog (2024-04-13 12:09:41)

Trilby · 2024-04-12 12:53:16

Python? No, no, no. This is clearly a job for a Java applet, or perhaps Visual Basic ... or Fortran.

seth · 2024-04-12 14:07:25

Those are all unsafe options.
It needs to be re-implemented in rust

dakota · 2024-04-12 19:07:07

Isn't there an emacs command to do that?

seth · 2024-04-12 22:15:19

cmd+alt+l_arrow,=,r_arrow,?

Arch Linux

#1 2024-04-09 02:56:51

search text file for lines present in another text file and redirect.

#2 2024-04-09 03:43:54

Re: search text file for lines present in another text file and redirect.

#3 2024-04-09 04:00:50

Re: search text file for lines present in another text file and redirect.

#4 2024-04-09 04:44:23

Re: search text file for lines present in another text file and redirect.

#5 2024-04-09 09:19:03

Re: search text file for lines present in another text file and redirect.

#6 2024-04-09 12:50:33

Re: search text file for lines present in another text file and redirect.

#7 2024-04-09 14:54:27

Re: search text file for lines present in another text file and redirect.

#8 2024-04-09 15:29:54

Re: search text file for lines present in another text file and redirect.

#9 2024-04-09 15:32:18

Re: search text file for lines present in another text file and redirect.

#10 2024-04-09 16:13:34

Re: search text file for lines present in another text file and redirect.

#11 2024-04-09 16:36:39

Re: search text file for lines present in another text file and redirect.

#12 2024-04-09 18:07:25

Re: search text file for lines present in another text file and redirect.

#13 2024-04-10 01:32:18

Re: search text file for lines present in another text file and redirect.

#14 2024-04-10 07:29:29

Re: search text file for lines present in another text file and redirect.

#15 2024-04-10 12:29:46

Re: search text file for lines present in another text file and redirect.

#16 2024-04-10 13:45:45

Re: search text file for lines present in another text file and redirect.

#17 2024-04-10 13:52:16

Re: search text file for lines present in another text file and redirect.

#18 2024-04-10 14:51:41

Re: search text file for lines present in another text file and redirect.

#19 2024-04-10 16:51:13

Re: search text file for lines present in another text file and redirect.

#20 2024-04-10 19:05:20

Re: search text file for lines present in another text file and redirect.

#21 2024-04-12 08:35:37

Re: search text file for lines present in another text file and redirect.

#22 2024-04-12 12:53:16

Re: search text file for lines present in another text file and redirect.

#23 2024-04-12 14:07:25

Re: search text file for lines present in another text file and redirect.

#24 2024-04-12 19:07:07

Re: search text file for lines present in another text file and redirect.

#25 2024-04-12 22:15:19

Re: search text file for lines present in another text file and redirect.

Board footer