SBFspot Upload Performance

[tonybroadbent]

The SpotData table is storing the TimeStamp as an Int(4).
This is then converted to a proper timestamp in the views, using From_UnixTime.
The problem with this approach is that queries can't use the index, and have to scan the entire table. OK initially, but on my little pi (with 100,000 records) it's taking 5 seconds to run the query to find records to upload:

SELECT DATE_FORMAT(Timestamp,'%Y%m%d,%H:%i'),V1,V2,V3,V4,V5,V6,V7,V8,V9,V10,V11,V12
FROM vwPvoData 
WHERE TimeStamp>NOW()-INTERVAL 13 DAY 
AND PVoutput IS NULL 
AND Serial=99999999 
ORDER BY TimeStamp LIMIT 30;

There is a clone of this repository at
https://github.com/warthog618/SBFspot/blob/master/SBFspot/CreateSQLiteDB.sql
where warthog618 attempts to address this problem by storing Timestamp as a datetime.
It can be turned back into a number in views if required.

The problem for me is the database is slowing down my whole PI, causing timeouts to other workloads.
Thanks,
Tony

[sbf-]

You're running your MySQL/MariaDB on the same Rpi I suppose? I don't recommend this.
I'm aware of this warthog clone. Since the very beginning I heard only a few complaints of poor performance, so I didn't look at it thoroughly.
Time is stored as unixtime which fits in an int(4) - This is the format used by the inverter.

[prein2]

Same problem here: raspberry PI3, uploaddeamon is causing 100% CPU time.
SQlite DB file is 73MB.
So possible solution:

• something like warthog618 did (tried that branche, but it is very old, gives errors in new version)
• can we clear old data, say everything except last month or so. I don't need te old data since it is in PVoutput.

[mr-russ]

I'm looking to investigate some things about performance and uploading consumption. Is anybody willing to supply me a slow database for me to investigate on?

[mr-russ]

Also looking at the database structure, an index on PVoutput will provide significant gains if the query optimizer can work out that very few values of PVoutput are null.

[notreallybob]

I have this slow sqlite query too. The performance is steadily getting slower as the number of records in DayData and SpotData tables increase. They currently have around 180,000 records each.
The slowness is in function db_SQL_Base::batch_get_archdaydata(). I have managed to cut the query time down from 30+ seconds to around 14 seconds by modifying the sql statement.
If SBFspot author, or anyone else is interested, here is what I did.
In file SBFspot/db_SQLite.cpp, around line 230, replace the sql line with this:

    sql <<
        "SELECT strftime('%Y%m%d,%H:%M', dd.TimeStamp) as TimeStamp,"
            "dd.TotalYield AS V1,"                                                                                            
            "dd.Power AS V2,"
            "cons.EnergyUsed AS V3,"                                                                                          
            "cons.PowerUsed AS V4,"
            "spot.Temperature AS V5,"
            "spot.Uac1 AS V6,"                                                                                                
            "NULL AS V7,"                                                                                                     
            "NULL AS V8,"
            "NULL AS V9,"
            "NULL AS V10,"
            "NULL AS V11,"
            "NULL AS V12 "
        //--FROM vwDayData AS dd
        "FROM ("                                                                                                              
        "    SELECT datetime(Dat.TimeStamp, 'unixepoch', 'localtime') AS TimeStamp,"                                          
        "       Inv.Name,"                                                                                                    
        "       Inv.Type,"
        "       Dat.Serial,"                                                                                                  
        "       TotalYield,"                                                                                                  
        "       Power,"                                                                                                       
        "       PVOutput"
        "    FROM     DayData Dat"                                                                                            
        "    JOIN Inverters Inv ON Dat.Serial = Inv.Serial"                                                                   
        "    where datetime(Dat.TimeStamp, 'unixepoch', 'localtime') > DATE(DATE(),'" << -(datelimit-2) << " day') "          
        "    and   pvoutput is null"                                                                                          
        "    AND   dat.Serial = " << Serial << " "                                                                            
        ") as dd "

        //--LEFT JOIN vwAvgSpotData    AS spot ON dd.Serial    = spot.Serial AND dd.Timestamp = spot.Nearest5min
        "left join ("
        "    SELECT  Dat.Serial,"
        "            datetime("
        "                        CASE"
        "                            WHEN (Dat.TimeStamp % 300) < 150 THEN Dat.TimeStamp - (Dat.TimeStamp % 300)"
        "                            ELSE Dat.TimeStamp - (Dat.TimeStamp % 300) + 300"
        "                        END,"
        "                        'unixepoch', 'localtime'"
        "                    ) AS Nearest5min,"
        "           avg(Uac1) as Uac1,"
        "           avg(ROUND(Dat.Temperature, 1)) AS Temperature"
        "    FROM SpotData Dat"
        "    INNER JOIN Inverters Inv ON Dat.Serial = Inv.Serial"
        "    where Dat.Serial = " << Serial << " "
        "    group by 1,2"
        ") as spot on spot.Serial = dd.Serial and spot.nearest5min = dd.Timestamp "

        "LEFT JOIN vwAvgConsumption AS cons ON dd.Timestamp = cons.Nearest5min "

        /*
        "WHERE   dd.TimeStamp > DATE(DATE(),'" << -(datelimit-2) << " day') "
        "AND     PVoutput IS NULL "
        "AND     dd.Serial=" << Serial << " "
        */

        "ORDER BY dd.Timestamp "
        "LIMIT " << statuslimit;

[notreallybob]

Made it faster still, by using the date filter for the SpotData subquery.
30 seconds down to 6 seconds.

    sql <<
        "SELECT strftime('%Y%m%d,%H:%M', dd.TimeStamp) as TimeStamp,"
            "dd.TotalYield AS V1,"
            "dd.Power AS V2,"
            "cons.EnergyUsed AS V3,"
            "cons.PowerUsed AS V4,"
            "spot.Temperature AS V5,"
            "spot.Uac1 AS V6,"
            "NULL AS V7,"
            "NULL AS V8,"
            "NULL AS V9,"
            "NULL AS V10,"
            "NULL AS V11,"
            "NULL AS V12 "
        //--FROM vwDayData AS dd
        "FROM ("
        "    SELECT datetime(Dat.TimeStamp, 'unixepoch', 'localtime') AS TimeStamp,"
        "       Inv.Name,"
        "       Inv.Type,"
        "       Dat.Serial,"
        "       TotalYield,"
        "       Power,"
        "       PVOutput"
        "    FROM     DayData Dat"
        "    JOIN Inverters Inv ON Dat.Serial = Inv.Serial"
        "    where datetime(Dat.TimeStamp, 'unixepoch', 'localtime') > DATE(DATE(),'" << -(datelimit-2) << " day') "
        "    and   pvoutput is null"
        "    AND   dat.Serial = " << Serial << " "
        ") as dd "

        //--LEFT JOIN vwAvgSpotData    AS spot ON dd.Serial    = spot.Serial AND dd.Timestamp = spot.Nearest5min
        "left join ("
        "    SELECT  Dat.Serial,"
        "            datetime("
        "                        CASE"
        "                            WHEN (Dat.TimeStamp % 300) < 150 THEN Dat.TimeStamp - (Dat.TimeStamp % 300)"
        "                            ELSE Dat.TimeStamp - (Dat.TimeStamp % 300) + 300"
        "                        END,"
        "                        'unixepoch', 'localtime'"
        "                    ) AS Nearest5min,"
        "           avg(Uac1) as Uac1,"
        "           avg(ROUND(Dat.Temperature, 1)) AS Temperature"
        "    FROM SpotData Dat"
        "    INNER JOIN Inverters Inv ON Dat.Serial = Inv.Serial"
        "    where Dat.Serial = " << Serial << " "
        "    and   datetime(Dat.TimeStamp, 'unixepoch', 'localtime') > DATE(DATE(),'" << -(datelimit-2) << " day') "
        "    group by 1,2"
        ") as spot on spot.Serial = dd.Serial and spot.nearest5min = dd.Timestamp "

        "LEFT JOIN vwAvgConsumption AS cons ON dd.Timestamp = cons.Nearest5min "

        /*
        "WHERE   dd.TimeStamp > DATE(DATE(),'" << -(datelimit-2) << " day') "
        "AND     PVoutput IS NULL "
        "AND     dd.Serial=" << Serial << " "
        */

        "ORDER BY dd.Timestamp "
        "LIMIT " << statuslimit;

[sbf-]

Seriously, I don’t get it why you put all this SQL code inside the source instead of the view. Anyway nice job already. I’ll test it one of these days.

[notreallybob]

Views don't get optimised very well, hence inlining was required.

[sbf-]

Views don't get optimised very well, hence inlining was required.

This statement is wrong. I did the test and there is no notable difference in speed.

I digged up the reviewed queries from SBFspot 4 project which was put on hold (lack of time)
I combined it all together and this is the result for SQLite:

Running SBFspot V3.6.0 on Raspberry Pi 3 Model B Rev 1.2
Size SBFspot.db = 65MB
DayData : 292496 records
SpotData: 287771 records


Average query time (0 or 1 records available) : 0.108 sec
Average query time (batch of 30 records)      : 0.203 sec

There are some new views created for generation. Consumption data is not necessarily available at the same time of generation. So a separate upload is preferable.

--
-- vwPVODayData View
--
DROP VIEW IF EXISTS vwPVODayData;

CREATE VIEW vwPVODayData AS
       SELECT TimeStamp,
              Serial,
              TotalYield,
              Power
         FROM DayData Dat
        WHERE TimeStamp > strftime( '%s', 'now' ) -( 
                  SELECT Value
                    FROM Config
                   WHERE [KEY] = 'Batch_DateLimit' 
              ) 
              * 86400 
              AND
              PvOutput IS NULL;

--
-- vwPVOSpotData View
--
DROP VIEW IF EXISTS vwPVOSpotData;

CREATE VIEW vwPVOSpotData AS
       SELECT TimeStamp,
              TimeStamp -( TimeStamp % 300 ) AS Nearest5min,
              Serial,
              Pdc1,
              Pdc2,
              Idc1,
              Idc2,
              Udc1,
              Udc2,
              Pac1,
              Pac2,
              Pac3,
              Iac1,
              Iac2,
              Iac3,
              Uac1,
              Uac2,
              Uac3,
              Pdc1 + Pdc2 AS PdcTot,
              Pac1 + Pac2 + Pac3 AS PacTot,
              ROUND( Temperature, 1 ) AS Temperature
         FROM SpotData
        WHERE TimeStamp > strftime( '%s', 'now' ) -( 
                  SELECT Value
                    FROM Config
                   WHERE [KEY] = 'Batch_DateLimit' 
              ) 
              * 86400;

--
-- vwPVOSpotDataAvg View
--
DROP VIEW IF EXISTS vwPVOSpotDataAvg;

CREATE VIEW vwPVOSpotDataAvg AS
       SELECT nearest5min,
              serial,
              avg( Pdc1 ) AS Pdc1,
              avg( Pdc2 ) AS Pdc2,
              avg( Idc1 ) AS Idc1,
              avg( Idc2 ) AS Idc2,
              avg( Udc1 ) AS Udc1,
              avg( Udc2 ) AS Udc2,
              avg( Pac1 ) AS Pac1,
              avg( Pac2 ) AS Pac2,
              avg( Pac3 ) AS Pac3,
              avg( Iac1 ) AS Iac1,
              avg( Iac2 ) AS Iac2,
              avg( Iac3 ) AS Iac3,
              avg( Uac1 ) AS Uac1,
              avg( Uac2 ) AS Uac2,
              avg( Uac3 ) AS Uac3,
              avg( Temperature ) AS Temperature
         FROM vwPVOSpotData
        GROUP BY serial,
                 nearest5min;

--
-- vwPVOUploadGeneration View
--
DROP VIEW IF EXISTS vwPVOUploadGeneration;

CREATE VIEW vwPVOUploadGeneration AS
       SELECT datetime( dd.TimeStamp, 'unixepoch', 'localtime' ) AS TimeStamp,
              dd.Serial,
              dd.TotalYield AS V1,
              dd.Power AS V2,
              NULL AS V3,
              NULL AS V4,
              spot.Temperature AS V5,
              spot.Uac1 AS V6,
              NULL AS V7,
              NULL AS V8,
              NULL AS V9,
              NULL AS V10,
              NULL AS V11,
              NULL AS V12
         FROM vwPVODayData AS dd
              LEFT JOIN vwPVOSpotDataAvg AS spot
                     ON dd.Serial = spot.Serial 
                     AND
                     dd.Timestamp = spot.Nearest5min

This is the modification in db_SQLite.cpp

	sql << "SELECT strftime('%Y%m%d,%H:%M',TimeStamp),V1,V2,V3,V4,V5,V6,V7,V8,V9,V10,V11,V12 FROM [vwPvoUploadGeneration] WHERE "
		"Serial=" << Serial << " "
		"ORDER BY TimeStamp "
		"LIMIT " << statuslimit;

Here is the logging (I needed a new timestamp function to have millisecond resolution)

[2019-10-14 09:58:58.740] INFO: SBFspotUploadDaemon Version 2.0.0
[2019-10-14 09:58:58.747] INFO: Starting Daemon...
[2019-10-14 09:58:59.116] INFO: Starting batch_get_archdaydata()
[2019-10-14 09:58:59.314] INFO: Back from batch_get_archdaydata()
[2019-10-14 09:58:59.717] INFO: Uploading 30 datapoints, starting with 20191009,08:15,36568965,0,,,23.6,232.72 => OK (200)
[2019-10-14 10:00:30.707] INFO: Starting batch_get_archdaydata()
[2019-10-14 10:00:30.939] INFO: Back from batch_get_archdaydata()
[2019-10-14 10:00:31.361] INFO: Uploading 30 datapoints, starting with 20191009,10:45,36569252,228,,,33.8,234.43 => OK (200)
[2019-10-14 10:01:30.332] INFO: Starting batch_get_archdaydata()
[2019-10-14 10:01:30.583] INFO: Back from batch_get_archdaydata()
[2019-10-14 10:01:30.992] INFO: Uploading 30 datapoints, starting with 20191009,13:15,36571097,2400,,,37.5,237.98 => OK (200)
[2019-10-14 10:02:30.252] INFO: Starting batch_get_archdaydata()
[2019-10-14 10:02:30.414] INFO: Back from batch_get_archdaydata()
[2019-10-14 10:02:30.796] INFO: Uploading 30 datapoints, starting with 20191009,15:45,36573752,420,,,39.5,235.96 => OK (200)
[2019-10-14 10:03:30.777] INFO: Starting batch_get_archdaydata()
[2019-10-14 10:03:30.948] INFO: Back from batch_get_archdaydata()
[2019-10-14 10:03:31.224] INFO: Uploading 10 datapoints, starting with 20191009,18:15,36574980,156,,,36.5,235.7 => OK (200)
[2019-10-14 10:04:30.333] INFO: Starting batch_get_archdaydata()
[2019-10-14 10:04:30.435] INFO: Back from batch_get_archdaydata()
[2019-10-14 10:05:30.454] INFO: Starting batch_get_archdaydata()
[2019-10-14 10:05:30.570] INFO: Back from batch_get_archdaydata()
[2019-10-14 10:06:30.588] INFO: Starting batch_get_archdaydata()
[2019-10-14 10:06:30.682] INFO: Back from batch_get_archdaydata()
[2019-10-14 10:07:30.701] INFO: Starting batch_get_archdaydata()
[2019-10-14 10:07:30.813] INFO: Back from batch_get_archdaydata()
[2019-10-14 10:08:30.833] INFO: Starting batch_get_archdaydata()
[2019-10-14 10:08:30.945] INFO: Back from batch_get_archdaydata()
[2019-10-14 10:09:30.964] INFO: Starting batch_get_archdaydata()
[2019-10-14 10:09:31.075] INFO: Back from batch_get_archdaydata()
[2019-10-14 10:10:30.093] INFO: Starting batch_get_archdaydata()
[2019-10-14 10:10:30.210] INFO: Back from batch_get_archdaydata()
[2019-10-14 10:11:30.229] INFO: Starting batch_get_archdaydata()
[2019-10-14 10:11:30.327] INFO: Back from batch_get_archdaydata()
[2019-10-14 10:12:30.346] INFO: Starting batch_get_archdaydata()
[2019-10-14 10:12:30.461] INFO: Back from batch_get_archdaydata()
[2019-10-14 10:12:30.737] INFO: Uploading datapoint: 20191009,19:00,36575037,12,,,36,235.56 => OK (200)
[2019-10-14 10:13:30.440] INFO: Starting batch_get_archdaydata()
[2019-10-14 10:13:30.533] INFO: Back from batch_get_archdaydata()

[notreallybob]

I'm quite confident that my targeted sql is better than the sqlite's views query optimizer for this instance.
You may not have observed much improvement, but then again you've mentioned not seeing a performance problem to begin with. Ask someone else who has experienced poor performance to try it out. I know it has helped for my case, but I could just be an outlier.
In any case, it is your show so you get to choose which contributions to accept or not.

[sbf-]

I'm afraid I was not clear enough...
When I compare the original views with their replacement with tables only I don't see any difference. This makes me conclude that views don't slow down the querying, at least not visible to me.
When I compare the original views with your last SQL statement there is indeed a big improvement (at least 5 times faster)
I was able to optimize the query and the performance boosted from 30 to 0.1 sec
For readability I prefer the views approach. Again, I noticed no difference when using tables only.

[notreallybob]

Indeed, I misunderstood your earlier message. All understood now. thumbs up

[stripwax]

@sbf- Is your optimised query (0.1 seconds) committed already at this point? Previous comment ambiguous but issue still open so I think that means no?

[sbf-]

No, not yet

[lnu]

Be careful that the optimized views don't work(as it is) with multiple inverters.

[sbf-]

Be careful that the optimized views don't work(as it is) with multiple inverters.

Can you explain this a bit more?

[lnu]

I tried the changes(views and code) and the data uploaded in pvoutput contains the output of only one of the inverters(I have two inverters configured with mis_enabled). The average column in pvoutput is always equal to 0.

[RossWilliamson]

I would be looking for a MYSQL solution to this too. I can take a stab at it if the sqlite versions is baselined

[prein2]

Tried using this latest version, since I'm having performance issues after 6 months of running (with 3 inverters).

I'm also using a customised view to upload string data to PVOutput:

CREATE VIEW vwPvoData AS
    SELECT dd.Timestamp,
        dd.Name,
        dd.Type,
        dd.Serial,
        dd.TotalYield AS V1,
        dd.Power AS V2,
        cons.EnergyUsed AS V3,
        cons.PowerUsed AS V4,
        spot.Temperature AS V5,
        spot.Uac1 AS V6,
        spot.Udc1 AS V7,  -- DC Voltage String 1
        spot.Udc2 AS V8,  -- DC Voltage String 2
        spot.Pdc1 AS V9,  -- DC Power String 1
        spot.Pdc2 AS V10, -- DC Power String 2
        NULL AS V11,
        NULL AS V12,
        dd.PVoutput
    FROM vwDayData AS dd
    LEFT JOIN vwAvgSpotData AS spot ON dd.Serial = spot.Serial AND dd.Timestamp = spot.Nearest5min
    LEFT JOIN vwAvgConsumption AS cons ON dd.Timestamp = cons.Nearest5min
    ORDER BY dd.Timestamp DESC;

But since I'm using the new version v5-v10 do not get uploaded. The rest is working fine.
Any ideas?

[sbf-]

With "this latest version" you mean the "dev" branch?
Did you modify the vwPVOUploadGeneration view for V6-V12?

[prein2]

Yes, meant the dev branch.
I did not modify that view, followed the wiki where it says I only have to modify the vwPvoData view...

[sbf-]

It's a bit more complicated ;-)
Wiki is for current production release, for dev branch, you need these changes - See also Update_V3_SQLite.sql in dev branch
Before making changes to the db, take a backup of SBFspot.db

[qurm]

@sbf- Sorry, but can you clarify what the best approach is to apply this performance fix.

Per my question in discussions/479 I am using SBFspot V3.8.2 (main) with a fresh install, but an old database with ~140K records. (previously ran the Update_340_SQLite.sql, so I think the schema is correct for 3.8.2). I applied the above fix (create the new views, modify db_SQLite.cpp, re-compile the app and the Daemon) but that did not help. I now understand that this requires me to use the V4 dev branch?

I am happy to try the dev branch, but not clear if you are recommending it as ready for use? Does the basic functionality work correctly (single inverter, not measuring consumption, upload to pvoutput). I have a high level of technical knowledge, but don't want to start troubleshooting the C code :).
Thanks, Andy

[sbf-]

No need to use dev branch. It should work with current production version 3.8.2
Create the 4 views (vwPVODayData, vwPVOSpotData, vwPVOSpotDataAvg and vwPVOUploadGeneration)
Make the change in db_SQLite.cpp and rebuild the daemon
Stop daemon and SBFspot, copy new daemon to /usr/local/bin/sbfspot.3

Maybe a silly question, but are you sure you replaced the daemon with the new version? You should see a much better performance.
I have a spare Raspberry Pi Zero W Rev 1.1, so I'll retry the workaround in the next days.

[qurm]

I did check timestamps and the daemon seemed to have a new date. Does the service somehow cache the daemon (I have done a stop & start)?

Anyway I repeated the whole build process yesterday and it has run successfully today. Performance looks much better, and it does not take high CPU for long. I dont have any exact measure , but it appears on top for only 3-5 seconds with CPU of 90% then falling back to a few %. I have still seen some periods where it is slow to connect via SSH and WinSCP, but I think it is acceptable.

Maybe for a Pi Zero it would be worth reducing the interval - say every 10 minutes. Would you recommend a change to CommonServiceCode.cpp here?:

    // Wait for next run; 30 seconds after every 1 minute (08:00:30 - 08:01:30 - 08:02:30 - ...) 
    for (int countdown = 90 - (time(nullptr) % 60); !bStopping && countdown > 0; countdown--)
	sleep(1);

[sbf-]

Maybe for a Pi Zero it would be worth reducing the interval - say every 10 minutes. Would you recommend a change to CommonServiceCode.cpp here?:

Pi Zero has a single core processor vs quad core for Pi 3, that's a big difference (apart from clock speed)
Sure, you can lower the scanning interval to 5 or 10 minutes.

Every 5min
for (int countdown = 330 - (time(nullptr) % 300); !bStopping && countdown > 0; countdown--)

Every 10min
for (int countdown = 630 - (time(nullptr) % 600); !bStopping && countdown > 0; countdown--)

Maybe you can try to lower the daemon priority (see StackExchange)

I did the test myself on Pi Zero W (DB size 85Mb)
Original query: 135 seconds, no matter how many records to upload
New query: 5,990 seconds for 1021 records, 1,129 seconds for 30 records and 0,862 for only 1 record (which is the most common case)

[warthog618]

Hi, I'm the creator of the branch mentioned by the OP.
I've been running my branched version for years, in fact since before this repo even existed, but thought it was finally time to update my SBFspot installation to the current release to get fixes and other goodies like MQTT.
While the v3.8.2 SBFspotUploadDaemon is still slow, the view additions and db_SQLite.cpp changes suggested above work for me, and are even faster than the direct table approach, so I'll be using that from now on.
Are those changes in a branch yet?
Any plans for getting them into a release?
And how about making the scanning interval configurable?

[sbf-]

Any plans for getting them into a release?

Yes, Version 3.9

And how about making the scanning interval configurable?

Not a bad idea... I'll try to include it in the same release.

[FutureCow]

Is this upload performance patch included in the v3.9.3 release?

[sbf-]

Is this upload performance patch included in the v3.9.3 release?

Hmm, looks like I didn't keep my promise. I'll add it to the 3.10 milestone (coming soon)

[amokla]

Hi, I am also looking forward for this improvement. My Raspi 1B would be happy.

[janknieling]

I think supporting and using InfluxDB as backend (or TimescaleDB) can solve these problems too. The question is how much work is it to implement these changes like requested in #353 and #542

[mk-au]

Is there any update on this one? or something that can be done to improve the CPU time?

My Pi Zero W is almost running at 100% CPU all the time just uploading data to PVOutput.

[warthog618]

In addition to the changes suggested above, which will hopefully make it into v3.10, I added a Python script to my setup that I run weekly to archive old data into a separate database. That reduces the data in the upload database to the data for the last month or so, and keeps my Pi0 CPU usage to about 3-4%.
(archive script - https://gist.github.com/warthog618/db30df2567ec605439f77a09702b0962)

[glennodotcom]

For what its worth ive been hunting down performance bottlenecks on my Odroid M1 (which is a 4 core 64bit ARM based SBC). SBF Uploader consumes 100% of one CPU core for about 15 seconds every 5mins. During this time it writes ~55MB of data to disk for each inverter. This is on my sqlite database of ~35MB.

I will watch how this changes once 3.10 lands.

Edit: I was impatient so I decided to give the Python script here a whirl: https://gist.github.com/warthog618/db30df2567ec605439f77a09702b0962
My database shrunk to 2Mb, and my disk writes are ~32KB for both inverters every 5mins. The uploader process now takes about 1 second each 5mins. Massive difference to the CPU load!
It will be interesting to see how this creeps up over time. Ive been running SBFspot for ~18months now.