Logged data frequency in CSV

Really old thread, but did this ever get resolved?

Hey guys, The only update I have is that I logged a support ticket on 29/09/2015 and have only had one response from Justin (I would take it that this is a low priority for him - probably rightly so - just frustrating for some of us looking at small data samples - in my case - one specific lap). I was thinking this was either a problem with the extract to CSV routine however given some other comments here - and Justin's first comment on the ticket, it does sound like a hardware issue. From Justin: "This is not a problem with the conversion of the data to csv as the log time is stored in an hour, min, seconds, milliseconds format in the raw data log, so when a .wrl file is opened this is translated directly into the Log Timestamp. There are different rates of data from different ECU's so we sample to data from the various sources when there is new data available. The most likely cause is the internal clock on the micro controller in the device, can you estimate the amount the timestamp is out based on your experiments? FYI, the file timestamp uses a different mechanism to the data log timestamp which uses a high speed internal clock." I provided some more details on my analysis to Justin - just waiting for the next update. I'll keep you guys posted.

Re: Logged data frequency in CSV

A very simple suggestion: make a simulation using the range 12.0-15.0 for both filter AFR and average AFR. Remove any other filter and accept almost everything (large standard deviation etc.). Any value out of this AFR range does not buy you anything good as long as autotune is concerned.

Re: Logged data frequency in CSV

Any update on this? I am convinced that there's something weird going on with the timing of data logging. I don't have any problem with the time being linearly out, but I have been frustrated for a long time with the timing of the logged data. Sometimes the time delay in AFR change to a throttle input is exactly as expected (perhaps 50-130ms depending on exact circumstances), but other times the response in the AFR is actually instantaneous & in some cases even slightly before the throttle input (which is theoretically impossible if all data is being logged in sync). It's almost telepathic. If enough logs are fed into AT I understand that it'll even out, but I just started again from scratch after making mechanical changes & wanted to get in some AT sessions without having to ride for weeks at a time between tunes. I keep getting dodgy data from partial decelerations corrupting cells that otherwise had good data in them. This is despite having both transition filters set to 0.1. I have tried tightening the Cell AFR standard deviation value, but this is counter productive when there is more data in the cell from a part throttle decel than there is from an acceleration run. There is the IAP deceleration filter, but this makes no difference when tuning the TPS map. There's nothing to prevent decel data in the TPS map. I know this probably doesn't seem like the right thread for it, but I think that part of my frustration comes from a suspicion that the timing of the logged data is inconsistent or out slightly out of sync. This is all on a based on log box denso v2 on a K7 GSXR750. If anyone can shed any light on this it would be appreciated.

Re: Logged data frequency in CSV

Cool - thanks for checking that Dnico - a lot of effort just to double check this - much appreciated. At least now I know its not me going mad (it was a suspicion).

There must be either some internal clock problem (hope not) or a scaling error in the CSV extract routine (more likely). I've raised a case with Justin - hopefully he gets around to it soon.

Re: Logged data frequency in CSV

I also have a K7 750, so I thought I would do a very simple test to check what you have said. Ignition on & a stopwatch running pulled in the clutch & held in for exactly 60 seconds, then released for 30 seconds, then pulled in again for 30 seconds. I checked the log & from the time of having the clutch pulled in to releasing it showed 55 seconds in data viewer. In the two 30 second intervals after that it also lost about 2.5 seconds each. So in 1 minute real time it's only logging it as 55 seconds. My logbox is slower than real time. I also noticed that the clock is going out, but the weird thing is that it's going out in the opposite direction to how the logging time is out. The clocks time appears to be going faster than real time. It is 9 minutes & 15 seconds faster than the computer it was synchronised with 34 days ago.

Re: Logged data frequency in CSV

Cool. We're on the same page then Ballard.

I'll raise a support ticket with Justin as he might be too busy to see this post.

Re: Logged data frequency in CSV

You understood correctly, there might be a scaling factor issue in the logbox code and I suggested to fix it at the code level. As a workaround a post-processing fix is possible, but if there is a problem it is always better to fix the root cause.

Re: Logged data frequency in CSV

Thanks for the reply Ballard. Not sure what you mean - is this a config on the LogBox itself? or in WRT? Or in Dashware? I've looked everywhere and I cant find a setting like this. As Log Time is in seconds then the software is expecting seconds. I might just have to scale it myself - but it would be an approx at best. From below, if my real laptime was 1:57 (117s) and the logfile has the runtime for that lap of 1:46 (106s) then I guess the (assuming linear scaling) is just 117/106 = the logfile is running 1.104 times faster than reality. So I can just rescale the Log Time myself to adjust for this. I just hope it is linear and that this factor is consistent - across the file and across any other logs I will use. If the scale is non-linear and sporadically logged then Im in poo. I wouldnt mind heading from Justin on this... as I think a little tweak in the WRT extract routine would solve the problem. I reckon its just a rounding issue or something when taking whatever internal datatype hes storing the Log Time as and the conversion to the 00:00:00.000 format in the CSV. I say that because the logs are running FASTER than reality - as its impossible to log stuff in the future before its happened! Internally hes probably using a decimal datatype or some other weird and wonderful datatype - and then he being really nice and converting this to 00:00:00.000 format on extract and its just got a little hiccup in that formula. Thats my guess.

Re: Logged data frequency in CSV

It could be just a matter of setting the correct pace to the internal clock. Usually there is a "Thick per Second" constant (HZ), might be sufficient to decrease it.

Re: Logged data frequency in CSV

Hey guys, Well, it doesn't look like I have solved this problem with a custom interpolation. I loaded the data into SQLserver and did wrote some fancy SQL to interpolate the data back to a consistent 10Hz (I work with data for a living). Good enough for my video overlays at least. I assumed that the wildly varying samples per second was the root cause so normalising it back to 10Hz I thought would solve the problem. Alas it did not. What Im leaning towards now as the root cause is the timestamp does not line up with reality. ie: theres an increasing delay in the logging over time. Its not just a shift along the time scale (which I can adjust when syncing this data to the video and GPS data) but rather the logged data runs TOO FAST. That is, the WRT data creeps ahead of the video/GPS data over time. At Phillip Island, Im looking at about half the length of the straight out of sync ie: the WRT logged data has my clicking up into 6th (just before the bridge over the start/finish) whereas the video has me just entering turn 12. So its not looking like its the sporadic logging intervals at all thats causing the issue. Its hard to do much more accurate testing, however I did a sanity check on a lap. Assuming I change up into 6th at the same point on the straight each lap (pretty consistent) then the time between the two changes up into 6th comes out at 1:46 however for this lap I did a 1:57 (and from my sanity perspective there's no way I'm lapping in the 1:40's thats for sure!). Now something in me says that if the logging was stretched out rather than concatenated over time it might be something to do with slow logging or the like. However as its COMPRESSED the time (and Im going slower than the speed of light or thereabouts) then it should be impossible to log data before its happened (in this dimension anyway!). SOOOOO - heres my leap of logic - it must be something to do with the extract routine in the conversion to the CSV as Im sure whatever internal datatype you are using for the timestamp would be something native to the hardware/firmware you have created. Could you have a look into this for me Justin? I have a log file if you want to examine mine. I have checked a bunch of logfiles over the last year or so and they are exhibit this behaviour. Half of the reason I bought the LogBox was for the analytics side of things (I'm a nerd) as opposed to the ECU flashing. So I'd be gutted if I couldn't use the data at all.

Logged data frequency in CSV

Hey Justin, I've got a little question for you. I have the Denso Logbox Pro for my GSX-R750K7 which is working great. All the WRT stuff works as expected. One of the things I have been playing with though is using the logged data from a trackday to overlay on a video feed (along with RaceChrono GPS data). I've got it all working (using Dashware) but my syncing of the WRT CSV to the real-time video and GPS based data gets out of sync really quickly. I think I have narrowed it down to the frequency of the logged data - specifically to the number of logged records per second. Its really inconsistent as to the frequency over time. A bit of a pivot table in Excel shows me the depth of the situation. For example, here is a summary of the number of records I get for each second (I just randomly chose seconds 66 through to 76). Time(s), Datapoints 66 28 67 37 68 41 69 7 70 39 71 38 72 37 73 34 74 32 75 27 76 56 I think Dashware is having real issues interpolating such a wide frequency of datapoints to plot. My question is this: is this normal? Is this expected? Does my gixxer's ECU really process data on this time sporadic basis? Is this a WRT to CSV issue or does my WRT logfile actually contain this kind of spread and theres nothing I can do about it? If theres nothing that can be done then I might just have to interpolate this myself (probably have to load it into my database and run some SQL windowing functions to do the job - not going to be pretty). If there is any way to get a consistent output (ie: you already have an interpolating/smoothing algorithm that you want to throw at it so that on export to CSV I can just choose the frequency eg: 10Hz and it does the work for me that would be cool). I guess what I'm looking for is more consistency like my GPS data. When I set the GPS to 10Hz I'm guaranteed 10 readings per second, no more no less. I'm guessing this is my ECU doing all of this though and not WRT or the logbox. (BTW - I haven't tried the high-frequency logging which I think I can do - 100Hz from memory - I'm just working with the default rate as that should suffice for my needs. Probably something to check in the future though). Just let me know if this is a bug or just something I have to live with and sort it out myself. Thanks mate.

View Topic on the Woolich Racing Forums