Re: LOTW Upload awkward and slow
Posted: 31 Mar 2020, 11:36
by KD0ZV
IW3HMH wrote: ↑31 Mar 2020, 08:11
LOTW expressly discourage the upload of single QSO "as soon as they're made" because they have a file queue system that is not made for realtime uploads.
This is why uploads are collected into a single file and sent all together
Yes, I am aware of that. The other services we take advantage of Log4OMs automatic feature.
What I would like to see is dedicated LOTW icons on the toolbar for upload and download.
UPLOAD - Automatically select required QSO's and upload. Unless you feel a prompt saying "you have X amount of QSO's ready for upload, do you want to send now?"
DOWNLOAD - Automatically check last download date and download.
These could be one clicks.
On a slightly different note, importing a N1MM log and uploading to QRZ is a painfully slow process. Does it have to be that way? And yes I know there is a method to connect N1MM to Log4OM but that requires me to shut off CAT and run N1MM all the time to make that work if I understand correctly. I asked about this in another thread but did not get a response.
G4POP wrote: ↑31 Mar 2020, 09:31
Perhaps it would be preferable to elect to automatically upload the LOTW QSO's on program close, this requires zero effort by the busy user and does not appear to significantly slow program closure.
I think for a lot of people that works great. Actually most the guys I have helped setup Log4OM use that function and love it. However with my schedule (and the fact I UL/DL from LOTW several times a day) it would require me to close and reopen the program.. often.
G4POP wrote: ↑31 Mar 2020, 09:31
Of course each person can find significant drawbacks, to them, with aspects of any program if they try hard enough! it is not possible for a developer to please everyone.
That is certainly true. And certainly will refrain from constructive criticism if that is what you guys want. Log4OM v1 was already the best logging program out there. V2 is even better.
73,
Rich
kd0zv
Re: LOTW Upload awkward and slow
Posted: 31 Mar 2020, 13:38
by w9mdb
What application log do you want? I turned on the Trace log level and it doesn't really show the problem that I can see.
Also, what confused me was the hourglass which implies you can't do anything while it's displayed.
So now it takes about 20 seconds to do the processing...two queries happen....but at < 3 seconds each that still isn't the brunt of the CPU time apparently.
It appears that the LOTW query actually selects every record in the DB in order to then query the field "confirmations" inside the app. This is not the most efficient way of doing this of course. The confirmations field should be normalized to another table and the query result would in tens-of-milliseconds range then on a properly-indexed table.
If I run the same SELECT query myself via sqlite3 it takes under 3 seconds. When I run the QSL Manager Log4OMV2 goes to 100% CPU for over 15 seconds.
Run Time: real 2.887 user 2.453125 sys 0.437500
My database is on an SSD drive (but should be cached anyways so that shouldn't matter.
On an Intel Core I7-4702MQ 2.2Ghz
Here's the trace log
2020-03-31 13:31:00.8200 DEBUG: [FwFile][ComposeFilename] : ComposeUserFilename: Returning USER file C:\Users\mdbla\AppData\Roaming\Log4OM2\user\layout_datagrid_ManageConfirmation_user.json
2020-03-31 13:31:00.8346 DEBUG: [FwFile][JsonDeserializeObjectFromFile] : Deserializing C:\Users\mdbla\AppData\Roaming\Log4OM2\user\layout_datagrid_ManageConfirmation_user.json as L4ONG.BE.LAYOUT.DataGridViewStructure
2020-03-31 13:31:00.8512 DEBUG: [FwFile][LoadFile] : Begin load text file C:\Users\mdbla\AppData\Roaming\Log4OM2\user\layout_datagrid_ManageConfirmation_user.json
2020-03-31 13:31:00.8737 DEBUG: [FwFile][LoadFile] : Text file load completed C:\Users\mdbla\AppData\Roaming\Log4OM2\user\layout_datagrid_ManageConfirmation_user.json
2020-03-31 13:31:01.1861 TRACE: [#=zpCgxnbM1utAKFPKPqA==][MoveNext] : Executing planned scheduler...
2020-03-31 13:31:01.2037 TRACE: [#=zpCgxnbM1utAKFPKPqA==][MoveNext] : Planned scheduler executed...
2020-03-31 13:31:03.7340 TRACE: [DbSqlite][GetLog] : SELECT QsoId, Callsign, Band, Mode, QsoDate, Address, StationCallsign, DXCC, ArrlSect, QsoComplete, Age, AIndex, AntAz, AntEl, AntPath, Antenna, ArrlCheck, BandRX, CallsignUrl, Class, Cnty, Comment, Cont, ContactAssociations, ContactedOp, ContestId, Country, CQZone, Distance, EqCall, EMail, ForceInit, Freq, FreqRx, GridSquare, ITUZone, KIndex, Lat, Lon, MaxBursts, MSShower, MyAssociations, MyDxcc, MyLat, MyLon, MyCity, MyCnty, MyCountry, MyCQZone, MyGridSquare, MyITUZone, MyName, MyPostalCode, MyStreet, MyRig, MySig, MySigInfo, MyState, Name, Notes, NrBursts, NrPings, Pfx, Operator, OwnerCallsign, Precedence, PropMode, ProgramId, ProgramVersion, QslMsg, QslVia, QsoEndDate, QsoRandom, Qth, RxPwr, Sig, SigInfo, RSTRcvd, RSTSent, SatelliteQSO, SatMode, SatName, SFI, SRX, SRXString, State, STX, STXString, SWL, TxPwr, QsoConfirmations, ContactReferences, MyReferences FROM LOG ORDER BY log.QsoDate DESC
2020-03-31 13:31:06.2458 TRACE: [#=zpCgxnbM1utAKFPKPqA==][MoveNext] : Executing planned scheduler...
2020-03-31 13:31:06.2790 TRACE: [#=zpCgxnbM1utAKFPKPqA==][MoveNext] : Planned scheduler executed...
2020-03-31 13:31:06.8775 TRACE: [DbSqlite][GetLog] : SELECT QsoId, Callsign, Band, Mode, QsoDate, Address, StationCallsign, DXCC, ArrlSect, QsoComplete, Age, AIndex, AntAz, AntEl, AntPath, Antenna, ArrlCheck, BandRX, CallsignUrl, Class, Cnty, Comment, Cont, ContactAssociations, ContactedOp, ContestId, Country, CQZone, Distance, EqCall, EMail, ForceInit, Freq, FreqRx, GridSquare, ITUZone, KIndex, Lat, Lon, MaxBursts, MSShower, MyAssociations, MyDxcc, MyLat, MyLon, MyCity, MyCnty, MyCountry, MyCQZone, MyGridSquare, MyITUZone, MyName, MyPostalCode, MyStreet, MyRig, MySig, MySigInfo, MyState, Name, Notes, NrBursts, NrPings, Pfx, Operator, OwnerCallsign, Precedence, PropMode, ProgramId, ProgramVersion, QslMsg, QslVia, QsoEndDate, QsoRandom, Qth, RxPwr, Sig, SigInfo, RSTRcvd, RSTSent, SatelliteQSO, SatMode, SatName, SFI, SRX, SRXString, State, STX, STXString, SWL, TxPwr, QsoConfirmations, ContactReferences, MyReferences FROM LOG ORDER BY log.QsoDate DESC
2020-03-31 13:31:11.3264 TRACE: [#=zpCgxnbM1utAKFPKPqA==][MoveNext] : Executing planned scheduler...
2020-03-31 13:31:11.3479 TRACE: [#=zpCgxnbM1utAKFPKPqA==][MoveNext] : Planned scheduler executed...
2020-03-31 13:31:15.8310 TRACE: [UCDataGrid][#=zSDC3SetBFxJFfdELUSJ7Ymg=] : Setting colors
2020-03-31 13:31:16.4042 TRACE: [#=zpCgxnbM1utAKFPKPqA==][MoveNext] : Executing planned scheduler...
2020-03-31 13:31:16.4227 TRACE: [#=zpCgxnbM1utAKFPKPqA==][MoveNext] : Planned scheduler executed...
2020-03-31 13:31:21.4769 TRACE: [#=zpCgxnbM1utAKFPKPqA==][MoveNext] : Executing planned scheduler...
2020-03-31 13:31:21.5003 TRACE: [#=zpCgxnbM1utAKFPKPqA==][MoveNext] : Planned scheduler executed...
2020-03-31 13:31:26.5452 TRACE: [#=zpCgxnbM1utAKFPKPqA==][MoveNext] : Executing planned scheduler...
2020-03-31 13:31:26.5664 TRACE: [#=zpCgxnbM1utAKFPKPqA==][MoveNext] : Planned scheduler executed...
2020-03-31 13:31:31.6054 TRACE: [#=zpCgxnbM1utAKFPKPqA==][MoveNext] : Executing planned scheduler...
2020-03-31 13:31:31.6249 TRACE: [#=zpCgxnbM1utAKFPKPqA==][MoveNext] : Planned scheduler executed...
2020-03-31 13:31:36.6648 TRACE: [#=zpCgxnbM1utAKFPKPqA==][MoveNext] : Executing planned scheduler...
2020-03-31 13:31:36.6814 TRACE: [#=zpCgxnbM1utAKFPKPqA==][MoveNext] : Planned scheduler executed...
2020-03-31 13:31:41.7178 TRACE: [#=zpCgxnbM1utAKFPKPqA==][MoveNext] : Executing planned scheduler...
2020-03-31 13:31:41.7391 TRACE: [#=zpCgxnbM1utAKFPKPqA==][MoveNext] : Planned scheduler executed...
2020-03-31 13:31:46.8019 TRACE: [#=zpCgxnbM1utAKFPKPqA==][MoveNext] : Executing planned scheduler...
2020-03-31 13:31:46.8205 TRACE: [#=zpCgxnbM1utAKFPKPqA==][MoveNext] : Planned scheduler executed...
2020-03-31 13:31:49.6919 TRACE: [WindowPositions][AddLocation] : [WIN POSITION] Updating form location for #=z_VQ3xWq98ki93COv8hHdgDw=: #=z_VQ3xWq98ki93COv8hHdgDw= X:213 Y:178 WIDTH:905 HEIGHT:525
2020-03-31 13:31:49.7417 TRACE: [UCDataGrid][WriteDatagridViewSettings] : Saving layout for ManageConfirmation
2020-03-31 13:31:49.7769 DEBUG: [FwFile][JsonSerializeObject] : JSON Serializing C:\Users\mdbla\AppData\Roaming\Log4OM2\user\layout_datagrid_ManageConfirmation_user.json
2020-03-31 13:31:49.8003 DEBUG: [FwFile][WriteFile] : Begin writing text file C:\Users\mdbla\AppData\Roaming\Log4OM2\user\layout_datagrid_ManageConfirmation_user.json
2020-03-31 13:31:49.8224 DEBUG: [FwFile][WriteFile] : Text file write completed C:\Users\mdbla\AppData\Roaming\Log4OM2\user\layout_datagrid_ManageConfirmation_user.json
2020-03-31 13:31:49.8370 DEBUG: [FwFile][JsonSerializeObject] : Serialization completed C:\Users\mdbla\AppData\Roaming\Log4OM2\user\layout_datagrid_ManageConfirmation_user.json
2020-03-31 13:31:51.8599 TRACE: [#=zpCgxnbM1utAKFPKPqA==][MoveNext] : Executing planned scheduler...
2020-03-31 13:31:51.8772 TRACE: [#=zpCgxnbM1utAKFPKPqA==][MoveNext] : Planned scheduler executed...