LOTW Upload awkward and slow

General discussions V2
w9mdb
Old Man
Posts: 434
Joined: 13 Jul 2014, 12:05

LOTW Upload awkward and slow

Post by w9mdb »

The most frequent use of QSL Manager for me is LOTW upload/download.
Using 2.4.0.0
I have 34,000 QSOs in my log.

The sequence to do it is not very fast.

#1 Click on Utilities/QSL Manager -- takes 14 seconds until the 1st refresh is done.
#2 Select the LOTW entry -- another 20 seconds -- it should remember this setting so that #1 is not necessary
#3 Click Select required -- another 20 seconds -- this SQL Query should be lightning faster with just a few records that should match an indexed field for LOTW status.
So this is 54 seconds when what should happen is this
#1 Click on Utilities/QSL Manager -- a couple seconds to retrieve LOTW Required entries....done...

As it is this discourages people from LOTW uploads since it takes so long.

de Mike W9MDB
User avatar
KD0ZV
Log4OM Alpha Team
Posts: 730
Joined: 04 Mar 2015, 13:42
Location: Ankeny, Iowa
Contact:

Re: LOTW Upload awkward and slow

Post by KD0ZV »

Hey Mike,

Your times seems excessive to me, but I agree 100% with what you are saying. Its a cumbersome process. My refreshes are closer to 2-3 seconds between steps and after practice I have the time down to about 20 seconds (on a good day) total. You have 3x the contacts I do though.

Since all the other methods like QRZ, Clublog, etc are automatic I would like to see the option to have a LOTW upload and download button on the toolbar.

As much as I like V2, this process was a step backwards for me.

BTW, I downloaded your fudge program last night to mess with my clock so I could work a Chatham Island contact on FT8 whose clock was way off and was having trouble getting him. Thanks!

Rich
73,
Rich
KD0ZV
Log4OM Alpha Team
User avatar
IW3HMH
Site Admin
Posts: 2925
Joined: 21 Jan 2013, 14:20
Location: Quarto d'Altino - Venezia (ITA)
Contact:

Re: LOTW Upload awkward and slow

Post by IW3HMH »

Hi
While i agree some features can (and will) be improved, i don't experience the same (excessively long) times in my system even with a test log composed by 150.000 QSO's (not proportional, at least)
About timings:

Search is automatically performed after each change. The search is triggered after 3 seconds from the last change applied in the search parameters to give you enough time to set all parameters one after each other. So when opening QSL screen you have 3 seconds wait, selecting LOTW another 3 seconds wait.
Each search takes 4 seconds on my system, so in 18 seconds i'm able to view LOTW data already selected for upload (with select required) using a 29.3k QSO log

Changing to manual search button can greatly reduce this time (i'm evaluating that option), but 20 seconds to show your QSO is definitely something related to location of your database file.
As asked several times, without an application log we cannot investigate further

LOTW expressly discourage the upload of single QSO "as soon as they're made" because they have a file queue system that is not made for realtime uploads.
This is why uploads are collected into a single file and sent all together
Daniele Pistollato - IW3HMH
User avatar
G4POP
Log4OM Alpha Team
Posts: 10752
Joined: 21 Jan 2013, 14:55
Location: Burnham on Crouch, Essex UK

Re: LOTW Upload awkward and slow

Post by G4POP »

Perhaps it would be preferable to elect to automatically upload the LOTW QSO's on program close, this requires zero effort by the busy user and does not appear to significantly slow program closure.

Of course each person can find significant drawbacks, to them, with aspects of any program if they try hard enough! it is not possible for a developer to please everyone.
73 Terry G4POP
User avatar
KD0ZV
Log4OM Alpha Team
Posts: 730
Joined: 04 Mar 2015, 13:42
Location: Ankeny, Iowa
Contact:

Re: LOTW Upload awkward and slow

Post by KD0ZV »

IW3HMH wrote: 31 Mar 2020, 08:11 LOTW expressly discourage the upload of single QSO "as soon as they're made" because they have a file queue system that is not made for realtime uploads.
This is why uploads are collected into a single file and sent all together
Yes, I am aware of that. The other services we take advantage of Log4OMs automatic feature.

What I would like to see is dedicated LOTW icons on the toolbar for upload and download.

UPLOAD - Automatically select required QSO's and upload. Unless you feel a prompt saying "you have X amount of QSO's ready for upload, do you want to send now?"
DOWNLOAD - Automatically check last download date and download.

These could be one clicks.

On a slightly different note, importing a N1MM log and uploading to QRZ is a painfully slow process. Does it have to be that way? And yes I know there is a method to connect N1MM to Log4OM but that requires me to shut off CAT and run N1MM all the time to make that work if I understand correctly. I asked about this in another thread but did not get a response.

G4POP wrote: 31 Mar 2020, 09:31 Perhaps it would be preferable to elect to automatically upload the LOTW QSO's on program close, this requires zero effort by the busy user and does not appear to significantly slow program closure.
I think for a lot of people that works great. Actually most the guys I have helped setup Log4OM use that function and love it. However with my schedule (and the fact I UL/DL from LOTW several times a day) it would require me to close and reopen the program.. often.
G4POP wrote: 31 Mar 2020, 09:31 Of course each person can find significant drawbacks, to them, with aspects of any program if they try hard enough! it is not possible for a developer to please everyone.
That is certainly true. And certainly will refrain from constructive criticism if that is what you guys want. Log4OM v1 was already the best logging program out there. V2 is even better.

73,

Rich
kd0zv
73,
Rich
KD0ZV
Log4OM Alpha Team
User avatar
G4POP
Log4OM Alpha Team
Posts: 10752
Joined: 21 Jan 2013, 14:55
Location: Burnham on Crouch, Essex UK

Re: LOTW Upload awkward and slow

Post by G4POP »

And certainly will refrain from constructive criticism if that is what you guys want.
We welcome constructive criticism but need to be mindful that getting side tracked fulfilling the specific requirements of one or two users takes up valuable time that needs to be spent resolving more global issues for the benefit of the user majority.

So keep the comments coming :-)
73 Terry G4POP
w9mdb
Old Man
Posts: 434
Joined: 13 Jul 2014, 12:05

Re: LOTW Upload awkward and slow

Post by w9mdb »

What application log do you want? I turned on the Trace log level and it doesn't really show the problem that I can see.
Also, what confused me was the hourglass which implies you can't do anything while it's displayed.
So now it takes about 20 seconds to do the processing...two queries happen....but at < 3 seconds each that still isn't the brunt of the CPU time apparently.

It appears that the LOTW query actually selects every record in the DB in order to then query the field "confirmations" inside the app. This is not the most efficient way of doing this of course. The confirmations field should be normalized to another table and the query result would in tens-of-milliseconds range then on a properly-indexed table.

If I run the same SELECT query myself via sqlite3 it takes under 3 seconds. When I run the QSL Manager Log4OMV2 goes to 100% CPU for over 15 seconds.
Run Time: real 2.887 user 2.453125 sys 0.437500
My database is on an SSD drive (but should be cached anyways so that shouldn't matter.
On an Intel Core I7-4702MQ 2.2Ghz

Here's the trace log
2020-03-31 13:31:00.8200 DEBUG: [FwFile][ComposeFilename] : ComposeUserFilename: Returning USER file C:\Users\mdbla\AppData\Roaming\Log4OM2\user\layout_datagrid_ManageConfirmation_user.json
2020-03-31 13:31:00.8346 DEBUG: [FwFile][JsonDeserializeObjectFromFile] : Deserializing C:\Users\mdbla\AppData\Roaming\Log4OM2\user\layout_datagrid_ManageConfirmation_user.json as L4ONG.BE.LAYOUT.DataGridViewStructure
2020-03-31 13:31:00.8512 DEBUG: [FwFile][LoadFile] : Begin load text file C:\Users\mdbla\AppData\Roaming\Log4OM2\user\layout_datagrid_ManageConfirmation_user.json
2020-03-31 13:31:00.8737 DEBUG: [FwFile][LoadFile] : Text file load completed C:\Users\mdbla\AppData\Roaming\Log4OM2\user\layout_datagrid_ManageConfirmation_user.json
2020-03-31 13:31:01.1861 TRACE: [#=zpCgxnbM1utAKFPKPqA==][MoveNext] : Executing planned scheduler...
2020-03-31 13:31:01.2037 TRACE: [#=zpCgxnbM1utAKFPKPqA==][MoveNext] : Planned scheduler executed...
2020-03-31 13:31:03.7340 TRACE: [DbSqlite][GetLog] : SELECT QsoId, Callsign, Band, Mode, QsoDate, Address, StationCallsign, DXCC, ArrlSect, QsoComplete, Age, AIndex, AntAz, AntEl, AntPath, Antenna, ArrlCheck, BandRX, CallsignUrl, Class, Cnty, Comment, Cont, ContactAssociations, ContactedOp, ContestId, Country, CQZone, Distance, EqCall, EMail, ForceInit, Freq, FreqRx, GridSquare, ITUZone, KIndex, Lat, Lon, MaxBursts, MSShower, MyAssociations, MyDxcc, MyLat, MyLon, MyCity, MyCnty, MyCountry, MyCQZone, MyGridSquare, MyITUZone, MyName, MyPostalCode, MyStreet, MyRig, MySig, MySigInfo, MyState, Name, Notes, NrBursts, NrPings, Pfx, Operator, OwnerCallsign, Precedence, PropMode, ProgramId, ProgramVersion, QslMsg, QslVia, QsoEndDate, QsoRandom, Qth, RxPwr, Sig, SigInfo, RSTRcvd, RSTSent, SatelliteQSO, SatMode, SatName, SFI, SRX, SRXString, State, STX, STXString, SWL, TxPwr, QsoConfirmations, ContactReferences, MyReferences FROM LOG ORDER BY log.QsoDate DESC
2020-03-31 13:31:06.2458 TRACE: [#=zpCgxnbM1utAKFPKPqA==][MoveNext] : Executing planned scheduler...
2020-03-31 13:31:06.2790 TRACE: [#=zpCgxnbM1utAKFPKPqA==][MoveNext] : Planned scheduler executed...
2020-03-31 13:31:06.8775 TRACE: [DbSqlite][GetLog] : SELECT QsoId, Callsign, Band, Mode, QsoDate, Address, StationCallsign, DXCC, ArrlSect, QsoComplete, Age, AIndex, AntAz, AntEl, AntPath, Antenna, ArrlCheck, BandRX, CallsignUrl, Class, Cnty, Comment, Cont, ContactAssociations, ContactedOp, ContestId, Country, CQZone, Distance, EqCall, EMail, ForceInit, Freq, FreqRx, GridSquare, ITUZone, KIndex, Lat, Lon, MaxBursts, MSShower, MyAssociations, MyDxcc, MyLat, MyLon, MyCity, MyCnty, MyCountry, MyCQZone, MyGridSquare, MyITUZone, MyName, MyPostalCode, MyStreet, MyRig, MySig, MySigInfo, MyState, Name, Notes, NrBursts, NrPings, Pfx, Operator, OwnerCallsign, Precedence, PropMode, ProgramId, ProgramVersion, QslMsg, QslVia, QsoEndDate, QsoRandom, Qth, RxPwr, Sig, SigInfo, RSTRcvd, RSTSent, SatelliteQSO, SatMode, SatName, SFI, SRX, SRXString, State, STX, STXString, SWL, TxPwr, QsoConfirmations, ContactReferences, MyReferences FROM LOG ORDER BY log.QsoDate DESC
2020-03-31 13:31:11.3264 TRACE: [#=zpCgxnbM1utAKFPKPqA==][MoveNext] : Executing planned scheduler...
2020-03-31 13:31:11.3479 TRACE: [#=zpCgxnbM1utAKFPKPqA==][MoveNext] : Planned scheduler executed...
2020-03-31 13:31:15.8310 TRACE: [UCDataGrid][#=zSDC3SetBFxJFfdELUSJ7Ymg=] : Setting colors
2020-03-31 13:31:16.4042 TRACE: [#=zpCgxnbM1utAKFPKPqA==][MoveNext] : Executing planned scheduler...
2020-03-31 13:31:16.4227 TRACE: [#=zpCgxnbM1utAKFPKPqA==][MoveNext] : Planned scheduler executed...
2020-03-31 13:31:21.4769 TRACE: [#=zpCgxnbM1utAKFPKPqA==][MoveNext] : Executing planned scheduler...
2020-03-31 13:31:21.5003 TRACE: [#=zpCgxnbM1utAKFPKPqA==][MoveNext] : Planned scheduler executed...
2020-03-31 13:31:26.5452 TRACE: [#=zpCgxnbM1utAKFPKPqA==][MoveNext] : Executing planned scheduler...
2020-03-31 13:31:26.5664 TRACE: [#=zpCgxnbM1utAKFPKPqA==][MoveNext] : Planned scheduler executed...
2020-03-31 13:31:31.6054 TRACE: [#=zpCgxnbM1utAKFPKPqA==][MoveNext] : Executing planned scheduler...
2020-03-31 13:31:31.6249 TRACE: [#=zpCgxnbM1utAKFPKPqA==][MoveNext] : Planned scheduler executed...
2020-03-31 13:31:36.6648 TRACE: [#=zpCgxnbM1utAKFPKPqA==][MoveNext] : Executing planned scheduler...
2020-03-31 13:31:36.6814 TRACE: [#=zpCgxnbM1utAKFPKPqA==][MoveNext] : Planned scheduler executed...
2020-03-31 13:31:41.7178 TRACE: [#=zpCgxnbM1utAKFPKPqA==][MoveNext] : Executing planned scheduler...
2020-03-31 13:31:41.7391 TRACE: [#=zpCgxnbM1utAKFPKPqA==][MoveNext] : Planned scheduler executed...
2020-03-31 13:31:46.8019 TRACE: [#=zpCgxnbM1utAKFPKPqA==][MoveNext] : Executing planned scheduler...
2020-03-31 13:31:46.8205 TRACE: [#=zpCgxnbM1utAKFPKPqA==][MoveNext] : Planned scheduler executed...
2020-03-31 13:31:49.6919 TRACE: [WindowPositions][AddLocation] : [WIN POSITION] Updating form location for #=z_VQ3xWq98ki93COv8hHdgDw=: #=z_VQ3xWq98ki93COv8hHdgDw= X:213 Y:178 WIDTH:905 HEIGHT:525
2020-03-31 13:31:49.7417 TRACE: [UCDataGrid][WriteDatagridViewSettings] : Saving layout for ManageConfirmation
2020-03-31 13:31:49.7769 DEBUG: [FwFile][JsonSerializeObject] : JSON Serializing C:\Users\mdbla\AppData\Roaming\Log4OM2\user\layout_datagrid_ManageConfirmation_user.json
2020-03-31 13:31:49.8003 DEBUG: [FwFile][WriteFile] : Begin writing text file C:\Users\mdbla\AppData\Roaming\Log4OM2\user\layout_datagrid_ManageConfirmation_user.json
2020-03-31 13:31:49.8224 DEBUG: [FwFile][WriteFile] : Text file write completed C:\Users\mdbla\AppData\Roaming\Log4OM2\user\layout_datagrid_ManageConfirmation_user.json
2020-03-31 13:31:49.8370 DEBUG: [FwFile][JsonSerializeObject] : Serialization completed C:\Users\mdbla\AppData\Roaming\Log4OM2\user\layout_datagrid_ManageConfirmation_user.json
2020-03-31 13:31:51.8599 TRACE: [#=zpCgxnbM1utAKFPKPqA==][MoveNext] : Executing planned scheduler...
2020-03-31 13:31:51.8772 TRACE: [#=zpCgxnbM1utAKFPKPqA==][MoveNext] : Planned scheduler executed...
User avatar
IW3HMH
Site Admin
Posts: 2925
Joined: 21 Jan 2013, 14:20
Location: Quarto d'Altino - Venezia (ITA)
Contact:

Re: LOTW Upload awkward and slow

Post by IW3HMH »

Log in trace mode makes application much slower, but it's not the case.
Records printed in the log are coming from different threads, so timings are not comparable one row over another.

What i can see is the sqlite access is sequential, so the 2 GetLog are roughly sequential.
As far as i can see the distance between 2 queries is:

2020-03-31 13:31:03.7340 TRACE: [DbSqlite][GetLog]
2020-03-31 13:31:06.8775 TRACE: [DbSqlite][GetLog]

I'm revamping the screen to introduce manual search button instead of automatic search after 3 seconds. This will save some time
Daniele Pistollato - IW3HMH
w9mdb
Old Man
Posts: 434
Joined: 13 Jul 2014, 12:05

Re: LOTW Upload awkward and slow

Post by w9mdb »

Yes...that matches the 2.8 second time I saw running the query myself. But that query result has, of course, all the QSOs and Log4OM gets very busy processing it for some reason.

So the question becomes why so long after query is done? No visibility there....

Mike
N6VH
Old Man
Posts: 186
Joined: 07 Nov 2015, 15:41

Re: LOTW Upload awkward and slow

Post by N6VH »

Add me to the list of those who think improvements could be made in the upload process. I think adding the manual search will certainly help. As has already been said, version 1 was very fast in the upload process.
Post Reply