node-tail prints out entire file, doesn't keep track of last know position in file

Question

node-tail prints out entire file, doesn't keep track of last know position in file

BestInSlot opened this issue 5 years ago · comments

Problem: I have a file being constantly written to, I'm using node-tail to watch this file and I'm piping the data through a socket to the front end. Unfortunately instead of receiving the latest appended information I'm receiving everything.

a very basic example (I'm not using a socket here, just console to log out the results):

const Tail = require("node-tail").Tail;
const tail = new Tail(file);

tail.on("line", data => console.log(data));

Luca Grulla · Answer 1 · Wed Aug 28 2019 04:02:59 GMT+0800 (China Standard Time)

Which OS?

When you say everything, you mean multiple line events, with each events reporting the whole content?

BestInSlot · Answer 2 · Wed Aug 28 2019 04:18:31 GMT+0800 (China Standard Time)

Lubuntu 18.04.

Happens when one line event is fired, but that line event will dump the entirety of the file. Instead of getting the appended data, I'm getting the old and new data dumped.

mostfurious · Answer 3 · Fri Nov 08 2019 16:21:43 GMT+0800 (China Standard Time)

When will this be resolved ? Does anyone know any other alternatives to use ? I tried "nodejs-tail", returs only ONE line and then does nothing, "tail-file" seems to have issues to even start, and this returns the WHOLE file conetent and not the single line.

Luca Grulla · Answer 4 · Sat Nov 09 2019 02:50:15 GMT+0800 (China Standard Time)

hi @mostfurious,

Are you experiencing the same problem?
Which OS are you using?

mostfurious · Answer 5 · Sat Nov 09 2019 03:14:27 GMT+0800 (China Standard Time)

Windows 10 build 1607 and Debian 9.
What i noticed is if you start with empty file and add one line or few it works fine
As in if you add one line it returns one line and if you add two lines it calls twice returning two lines..
But if you are warking with a file that already has some data in it lets say 50 lines, if you add a new one
it would call 51 times, add another one it will call 52 times and so on

Luca Grulla · Answer 6 · Sat Nov 09 2019 03:56:10 GMT+0800 (China Standard Time)

Same behaviour on Windows and Debian?

mostfurious · Answer 7 · Sat Nov 09 2019 04:24:41 GMT+0800 (China Standard Time)

I was working on Windows with empty files so i didnt notice any problems, i moved the code to Debian to filter logs and noticed that something was off, downloaded logs from Debian to Windows to check whats going on and then figured out that this is what was happening. That if the file has data in it, this plugin returns ALL file content not the last latest data.

Luca Grulla · Answer 8 · Sat Nov 09 2019 05:46:33 GMT+0800 (China Standard Time)

I cannot reproduce the scenario on MacOS X.
I could expect odd behaviours on Windows (some low level nodeJS functions are platform dependant), but I would expect consistent behaviour between MacOSX and Debian.

What the frequency of new lines addes to the file?

mostfurious · Answer 9 · Sat Nov 09 2019 15:41:55 GMT+0800 (China Standard Time)

Windows 10, v1607
Node version: v8.12.0

test.js content:

Tail = require('tail').Tail;
tail = new Tail("log.log");
tail.on("line", function(data) {
console.log(data);
});

log.log content before starting test.js:

127.2.0.1 - - [03/Nov/2019:09:16:40 +0200] "GET /wp-admins/js/customize-controls.minx.js?ver=7.2.5 HTTP/1.1" 200 28747
127.0.0.3 - - [03/Nov/2019:09:16:40 +0200] "GET /wp-admins/js/customize-controls.minx.js?ver=7.2.5 HTTP/1.1" 200 28747
127.3.0.1 - - [03/Nov/2019:09:16:40 +0200] "GET /wp-admins/js/customize-controls.minx.js?ver=7.2.5 HTTP/1.1" 200 28747

After the start i have log.log open and manualy copy/paste the last line "127.3.0.1 - - [03/Nov/2019:09:16:40 +0200] "GET /wp-admins/js/customize-controls.minx.js?ver=7.2.5 HTTP/1.1" 200 28747" and then press ctrl+s to save file.

Console output:

PS D:\Desktop\node> node .\test.js

127.3.0.1 - - [03/Nov/2019:09:16:40 +0200] "GET /wp-admins/js/customize-controls.minx.js?ver=7.2.5 HTTP/1.1" 200 28747127.2.0.1 - - [03/Nov/2019:09:16:40 +0200] "GET /wp-admins/js/customize-controls.minx.js?ver=7.2.5 HTTP/1.1" 200 28747
127.0.0.3 - - [03/Nov/2019:09:16:40 +0200] "GET /wp-admins/js/customize-controls.minx.js?ver=7.2.5 HTTP/1.1" 200 28747
127.3.0.1 - - [03/Nov/2019:09:16:40 +0200] "GET /wp-admins/js/customize-controls.minx.js?ver=7.2.5 HTTP/1.1" 200 28747
127.3.0.1 - - [03/Nov/2019:09:16:40 +0200] "GET /wp-admins/js/customize-controls.minx.js?ver=7.2.5 HTTP/1.1" 200 28747
127.3.0.1 - - [03/Nov/2019:09:16:40 +0200] "GET /wp-admins/js/customize-controls.minx.js?ver=7.2.5 HTTP/1.1" 200 28747127.2.0.1 - - [03/Nov/2019:09:16:40 +0200] "GET /wp-admins/js/customize-controls.minx.js?ver=7.2.5 HTTP/1.1" 200 28747
127.0.0.3 - - [03/Nov/2019:09:16:40 +0200] "GET /wp-admins/js/customize-controls.minx.js?ver=7.2.5 HTTP/1.1" 200 28747
127.3.0.1 - - [03/Nov/2019:09:16:40 +0200] "GET /wp-admins/js/customize-controls.minx.js?ver=7.2.5 HTTP/1.1" 200 28747
127.3.0.1 - - [03/Nov/2019:09:16:40 +0200] "GET /wp-admins/js/customize-controls.minx.js?ver=7.2.5 HTTP/1.1" 200 28747
127.3.0.1 - - [03/Nov/2019:09:16:40 +0200] "GET /wp-admins/js/customize-controls.minx.js?ver=7.2.5 HTTP/1.1" 200 28747

log.log content:

127.2.0.1 - - [03/Nov/2019:09:16:40 +0200] "GET /wp-admins/js/customize-controls.minx.js?ver=7.2.5 HTTP/1.1" 200 28747
127.0.0.3 - - [03/Nov/2019:09:16:40 +0200] "GET /wp-admins/js/customize-controls.minx.js?ver=7.2.5 HTTP/1.1" 200 28747
127.3.0.1 - - [03/Nov/2019:09:16:40 +0200] "GET /wp-admins/js/customize-controls.minx.js?ver=7.2.5 HTTP/1.1" 200 28747
127.3.0.1 - - [03/Nov/2019:09:16:40 +0200] "GET /wp-admins/js/customize-controls.minx.js?ver=7.2.5 HTTP/1.1" 200 28747
127.3.0.1 - - [03/Nov/2019:09:16:40 +0200] "GET /wp-admins/js/customize-controls.minx.js?ver=7.2.5 HTTP/1.1" 200 28747
127.3.0.1 - - [03/Nov/2019:09:16:40 +0200] "GET /wp-admins/js/customize-controls.minx.js?ver=7.2.5 HTTP/1.1" 200 28747

Debian 9
Node version: v12.13.0
Same script, same log, same procedure Putty output:

root@vps46:/node/node# node test.js
127.2.0.1 - - [03/Nov/2019:09:16:40 +0200] "GET /wp-admins/js/customize-controls.minx.js?ver=7.2.5 HTTP/1.1" 200 28747
127.0.0.3 - - [03/Nov/2019:09:16:40 +0200] "GET /wp-admins/js/customize-controls.minx.js?ver=7.2.5 HTTP/1.1" 200 28747
127.3.0.1 - - [03/Nov/2019:09:16:40 +0200] "GET /wp-admins/js/customize-controls.minx.js?ver=7.2.5 HTTP/1.1" 200 28747
127.3.0.1 - - [03/Nov/2019:09:16:40 +0200] "GET /wp-admins/js/customize-controls.minx.js?ver=7.2.5 HTTP/1.1" 200 28747127.2.0.1 - - [03/Nov/2019:09:16:40 +0200] "GET /wp-admins/js/customize-controls.minx.js?ver=7.2.5 HTTP/1.1" 200 28747
127.0.0.3 - - [03/Nov/2019:09:16:40 +0200] "GET /wp-admins/js/customize-controls.minx.js?ver=7.2.5 HTTP/1.1" 200 28747
127.3.0.1 - - [03/Nov/2019:09:16:40 +0200] "GET /wp-admins/js/customize-controls.minx.js?ver=7.2.5 HTTP/1.1" 200 28747
127.3.0.1 - - [03/Nov/2019:09:16:40 +0200] "GET /wp-admins/js/customize-controls.minx.js?ver=7.2.5 HTTP/1.1" 200 28747
127.3.0.1 - - [03/Nov/2019:09:16:40 +0200] "GET /wp-admins/js/customize-controls.minx.js?ver=7.2.5 HTTP/1.1" 200 28747127.2.0.1 - - [03/Nov/2019:09:16:40 +0200] "GET /wp-admins/js/customize-controls.minx.js?ver=7.2.5 HTTP/1.1" 200 28747
127.0.0.3 - - [03/Nov/2019:09:16:40 +0200] "GET /wp-admins/js/customize-controls.minx.js?ver=7.2.5 HTTP/1.1" 200 28747
127.3.0.1 - - [03/Nov/2019:09:16:40 +0200] "GET /wp-admins/js/customize-controls.minx.js?ver=7.2.5 HTTP/1.1" 200 28747
127.3.0.1 - - [03/Nov/2019:09:16:40 +0200] "GET /wp-admins/js/customize-controls.minx.js?ver=7.2.5 HTTP/1.1" 200 28747
127.3.0.1 - - [03/Nov/2019:09:16:40 +0200] "GET /wp-admins/js/customize-controls.minx.js?ver=7.2.5 HTTP/1.1" 200 28747

Sorry for the long post

Luca Grulla · Answer 10 · Sat Nov 09 2019 16:47:01 GMT+0800 (China Standard Time)

How have you appened your new line both on Windows and Debian)?

I tried and again I cannot reproduce the problem on MacOSX:

Tail = require('tail').Tail;
tail = new Tail("blah.txt",{logger: console}  );
tail.on("line", (data) =>{
  console.log(data);
});

echo "a1" >>blah.txt

Do you mind the same try the same approach to append on Debian?

By looking to the log it's also hard to really understand if it's repeating or is something else.
Is there a way for you to add a unique identifier to your log line so we can be 100% sure it's starting from the beginning every time? Right now they look all the same (minus IP rotating); It might be enough to introduce milliseconds in the timestamp to achieve the goal.

mostfurious · Answer 11 · Sat Nov 09 2019 18:02:04 GMT+0800 (China Standard Time)

Sorry for confusion,
On bot Windows and Debian i just open notepad press enter to make a new line then ctrl+v to paste the line and then ctrl+s to save file

Code:

Tail = require('tail').Tail;
tail = new Tail("log.log");

tail.on("line", function(data) {
console.log(${Date.now()} -- ${data});
});

log.log before:

New Line 1
New Line 2
New Line 3

I paste one and the same line at the time which is "New Line 3" and then save the file.

Windows output:

PS D:\Desktop\node> node .\test.js
1573292987096 --
1573292989766 -- New Line 3New Line 1
1573292989766 -- New Line 2
1573292989766 -- New Line 3
1573292989766 -- New Line 3
1573292995928 -- New Line 3New Line 1
1573292995928 -- New Line 2
1573292995928 -- New Line 3
1573292995928 -- New Line 3
1573292995928 -- New Line 3

How log.log looks after this:

New Line 1
New Line 2
New Line 3
New Line 3
New Line 3
New Line 3

Same steps on Debian 9 putty output:

root@vps46:/node/node# node test.js
1573293602856 -- New Line 1
1573293602861 -- New Line 2
1573293602861 -- New Line 3
1573293609362 -- New Line 3New Line 1
1573293609362 -- New Line 2
1573293609362 -- New Line 3
1573293609363 -- New Line 3
1573293616859 -- New Line 3New Line 1
1573293616860 -- New Line 2
1573293616860 -- New Line 3
1573293616861 -- New Line 3
1573293616862 -- New Line 3

Hope this clarifies things a bit

Luca Grulla · Answer 12 · Sat Nov 09 2019 18:28:04 GMT+0800 (China Standard Time)

I've seen notepad+Windows messing around in the past with operating system level file events in Node , so the test might be incorrect.

Rather then using Notepad can you please try:

echo newline >> log.log

to append a new line directly from your command line (both on Windows and Debian) and let me know?

mostfurious · Answer 13 · Sat Nov 09 2019 19:16:15 GMT+0800 (China Standard Time)

I guess the issue can be closed....
By using the command above in both Windows and Debian 9 i no longer get the whole file content but only last recent data.
I tried with live logs that are generated by openlitespeed web server and it seems to work fine now ?? Even tho it was giving me issues before.
For future reference i was adding lines to log.log with Notepad++ and then with Visual Studio code which gave me same results... I guess none of text editors can be used while working with this...
Sorry for wasting your time and thank you..

Luca Grulla · Answer 14 · Sun Nov 10 2019 00:24:29 GMT+0800 (China Standard Time)

glad to hear it's solved!

I won't close this because I have not had the time (yet) to verify what happens with Lubuntu.

ghita · Answer 15 · Mon Dec 30 2019 01:37:39 GMT+0800 (China Standard Time)

Have same behaviour as described in this bug description. On Windows 10.

Luca Grulla · Answer 16 · Mon Dec 30 2019 17:12:44 GMT+0800 (China Standard Time)

hi @ghita
Can you give more details?
Are you using an editor to add lines?

ghita · Answer 17 · Fri Mar 13 2020 15:22:34 GMT+0800 (China Standard Time)

Sorry for late reply. Yes, was using notepad++ to edit.
If I don't use that things work though.

Luca Grulla · Answer 18 · Fri Mar 13 2020 19:07:16 GMT+0800 (China Standard Time)

Great to hear @ghita.

GravityMsc · Answer 19 · Wed Jan 13 2021 16:13:57 GMT+0800 (China Standard Time)

@lucagrulla any plan to solve this problem? When I add a new line and save the file by using vscode, it still print all lines of the file.

Radosław Miluk · Answer 20 · Thu Sep 02 2021 06:28:52 GMT+0800 (China Standard Time)

I had the same issue on Windows. Used notepad for changing the content of a file instead of vscode, and that fixed the problem.

Clockworks · Answer 21 · Thu Sep 02 2021 06:30:30 GMT+0800 (China Standard Time)

I can second @VibingCreator
@lucagrulla please leave a note for people using the addon that using an external editor can cause this kind of behavior!
Perhaps you know a fix for it, but using notepad works fine (or writing to the file programmatically)

Best regards