Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Parsing a stream without clipping

I'm reading a stream, which is tested with a regex:

var deviceReadStream = fs.createReadStream("/path/to/stream");

deviceReadStream.on('data',function(data){
  if( data.match(aRegex) )
    //do something
});

But as the stream is splitted into several chuncks, it is possible that the cut make me miss a match. So there is a better pattern to test continuously a stream with a regex?

more details

The stream is the content of a crashed filesystem. I am searching for a ext2 signature (0xef53). As I do not know how the chunks are splitted, the signature could be splitted and not being detected.

So I used a loop to be able to delimite myself how the chunks are splitted, ie by block of the filesystem.

But using streams seems to be a better pattern, so how can I use streams while defining myself the chunks size ?

like image 828
Gaël Barbin Avatar asked Jul 22 '15 09:07

Gaël Barbin


People also ask

How do I make a clip on a stream?

If a streamer uses a mobile phone, it is possible to make a Clip there while he or she is broadcasting. You need to tap the screen while the stream is on, past broadcast or highlight you are watching;

Is it possible to stream without a capture card?

Pros of streaming without a capture card: This method doesn’t require a game capture card. You don’t need a powerful PC to be able to stream. There isn’t a lot of equipment requirements and the setup process is easy. This method can be used for any console as well as PC. Cons of streaming without a capture card:

How to create clips of your twitch streams?

That is why there are two more ways to create clips of your whole streams: 1. Connect YouTube account to Twitch If you are a channel owner, open the settings and select ‘Connections’ section. The next step is to enter ‘Past Broadcasts’, choose the video you need, press a gear wheel under the video and select ‘Export’;

Can you create a clip while streaming on YouTube?

As you have found out, it is very easy to create a Clip even if you are streaming at the moment. You can use not only a PC but also mobile devices for this purpose and take advantage of all the benefits you might get from shooting clips. Tania is a freelance writer and a social media fanatic.


2 Answers

Assuming your code just needs to search for the signature 0xef53 (as specified in the"more details" part of your question...

One way to do this and keep using regex is keep a reference to the previous data buffer, concatenate it with the current data buffer, and run the regex on that. Its a bit heavy on cpu usage since it effectively scans each data buffer twice (and there's lots of memory allocation due to the concatenation). It is relatively easy to read so it should be maintainable in the future.

Here's an example of what the code would look like

var deviceReadStream = fs.createReadStream("/path/to/stream");
var prevData = '';

deviceReadStream.on('data',function(data){
  var buffer = prevData + data;
  if( buffer.match(aRegex) )
    //do something

  prevData = data;
});

Another option would be to more manually do the character comparisons so the code can catch when the signature is split across data buffers. You can see a a solution to that in this related question Efficient way to search a stream for a string. According to the blog post of the top answer, the Haxe code he wrote can be built to produce JavaScript which you can then use. Or you could write your own custom code to do the search, since the signature that you're looking for is only 4 characters long.

like image 124
Ed Ballot Avatar answered Sep 18 '22 19:09

Ed Ballot


First, if you are determined to use a regex with nodejs, give pcre a try. A node wrapper for pcre is available. Pcre can be configured to do partial matches that can resume across buffer boundaries.

You might, though, just grep (or fgrep for multiple static strings) for a byte offset from the terminal. You can then follow it up with xxd and less to view it or dd to extract a portion.

For example, to get offsets with grep:

grep --text --byte-offset --only-matching --perl-regex "\xef\x53" recovery.img

Note that grep command line options can vary depending on your distro.

You could also look at bgrep though I haven't used it.

I have had good luck doing recovery using various shell tools and scripts.

A couple of other tangential comments:

  1. Keep in mind the endianness of whatever you are searching.
  2. Take an image since you are doing a recovery, if you have not already. Among other perils, if a device is starting to fail, further access can make it worse.
  3. Reference data carving tools. ref
  4. As you mentioned files may be fragmented. Still I would expect that partitions and files start on sector boundaries. As far as I know the magic would not typically be split.
  5. Be careful not to inadvertently write to the device you are recovering.
  6. As you may know, if you reconstruct the image you may be able to mount the image using a loopback driver.
like image 45
user650881 Avatar answered Sep 19 '22 19:09

user650881