LoTW weirdness

Need help? - Post here and we will find a solution for you.
NS8K
Advanced Class
Posts: 61
Joined: 29 Oct 2017, 12:54

LoTW weirdness

Post by NS8K »

I think I'm going crazy and cannot imagine what I've done. Starting on 12/12/17, when a QSO verification is downloaded from LoTW, the state field of the QSO record is being changed. This is a little hard to explain but the State field was IN before the verification download from LoTW and after the field contains IN // INDIANA. I am running JTAlert and WSJT-X and the original log record comes from JTAlert. This has been working beautifully and the log records that are created by JTAlert look just fine until a download from LoTW.

To test this a little further, I went into QSL Management and downloaded from LoTW going back one additional day to 12/11/17 and now, by George, all of my verifications from 12/11/17 on have the state field goofed up.

Here is what my 20m WAS status looks like
WAS.PNG
WAS.PNG (24.02 KiB) Viewed 7320 times
The county field has the same thing happening to it. If I log into LoTW and look at QSO detail, all of the state and county names look like this. I've never looked at that information before so i don't know if that is normal or a newly added "feature".

Anyway, does anybody else have this same trouble or is it just me? If just me, there must be something I have changed that would cause this odd behavior but I don't know what it is.
73 - Tom NS8K
NA5WH
Novice Class
Posts: 6
Joined: 14 Dec 2017, 05:13

Re: LoTW weirdness

Post by NA5WH »

I thought I had done something terribly wrong... I've been closely tracking my stats pages as I'm nearing WAS on 40m and 20m... and suddenly was like.. 59 of 59 states worked? whaaa? .... didn't know if its eqsl (don't 'download' that very often) or lotw... but popped up tonight. My guess is that lotw changed their adif file?...

edit: and I guess the next question is..... is there an easier way to correct the log... as I have a lot of lotws confirmed in the recent download...
ke5td
Novice Class
Posts: 16
Joined: 31 Aug 2016, 18:27
Location: Conway

Re: LoTW weirdness

Post by ke5td »

Tom,

Same problem here!!!! I corrected the state field in about 300 qso's and when I downloaded from LoTW the state field had all the errors back....definitely a problem from LoTW!!!!

Well maybe not ....I just checked my Ham Radio Deluxe log and it is not affected!!

Okay: from a much smarter man than me:

"Having said the above, this inclusion of "//" style comments should not break a properly coded adif parser as it should pass only the number of characters specified by the count included in the field name.
eg. passing "<STATE:2>MI // Michigan" will only return the first 2 characters, ie "MI" which is the correct result.

de Laurie VK3AMA"


This also affects the county field and the problem is the same.

While on the subject of state codes, why does Log4OM not use the standard "ME" instead of using "Me" ??
Last edited by ke5td on 14 Dec 2017, 10:29, edited 3 times in total.
User avatar
G4POP
Log4OM Alpha Team
Posts: 10752
Joined: 21 Jan 2013, 14:55
Location: Burnham on Crouch, Essex UK

Re: LoTW weirdness

Post by G4POP »

I have to ask why LOTW/ARRL keep going outside the standard ADIF format and thereby create issues for the users and developers?

Look at the tardiness they applied over the FT8 mode and many other examples prior to that, which is surprising considering they are a COMMERCIAL entity with paid staff!
73 Terry G4POP
NS8K
Advanced Class
Posts: 61
Joined: 29 Oct 2017, 12:54

Re: LoTW weirdness

Post by NS8K »

Thanks to everyone for chiming in. If I had known it was not me, I would have slept better last night! I happened to see the // in a few log entries but I have that column width narrowed down so it wasn't super obvious and I just thought humpf. It raised its ugly head when I later did a Scan Log and Rebuild in JTAlert and the WAS counts started going backward.

I have quite a few that are wrong but however this gets fixed, a re-download from LoTW for the affected dates will revert them back so I won't attempt any kind of manual correction.

Gary - I haven't noticed mixed case for state abbreviations. Where do you see this? Are you commenting specifically on Maine being Me or all states being mixed case? I'm curious.
73 - Tom NS8K
w9mdb
Old Man
Posts: 434
Joined: 13 Jul 2014, 12:05

Re: LoTW weirdness

Post by w9mdb »

If you download the sqlite3 command line utility you can fix this util LOTW or Log4OM fixes this.
Just run 'sqlite3 database.sqlite' with whatever your database name is and do this:
update log set state=SUBSTR(state,1,2);

de Mike W9MDB
ke5td
Novice Class
Posts: 16
Joined: 31 Aug 2016, 18:27
Location: Conway

Re: LoTW weirdness

Post by ke5td »

NS8K wrote: 14 Dec 2017, 13:51 Thanks to everyone for chiming in. If I had known it was not me, I would have slept better last night! I happened to see the // in a few log entries but I have that column width narrowed down so it wasn't super obvious and I just thought humpf. It raised its ugly head when I later did a Scan Log and Rebuild in JTAlert and the WAS counts started going backward.

I have quite a few that are wrong but however this gets fixed, a re-download from LoTW for the affected dates will revert them back so I won't attempt any kind of manual correction.

Gary - I haven't noticed mixed case for state abbreviations. Where do you see this? Are you commenting specifically on Maine being Me or all states being mixed case? I'm curious.

[I was referring to all state abbreviations ...when the qso is entered directly in Log4OM . If the qso is entered from JTAlert the state abbreviation is all upper case. Not a big deal just curious.... KE5TD/quote]
NS8K
Advanced Class
Posts: 61
Joined: 29 Oct 2017, 12:54

Re: LoTW weirdness

Post by NS8K »

w9mdb wrote: 14 Dec 2017, 16:34 If you download the sqlite3 command line utility you can fix this util LOTW or Log4OM fixes this.
Just run 'sqlite3 database.sqlite' with whatever your database name is and do this:
update log set state=SUBSTR(state,1,2);

de Mike W9MDB
Mike,

That was easy but yesterday I had a QSO with VK2DX in New South Wales. The state field is NSW // New South Wales which would become NS. Of course, the country would still be Australia. I think the NSW is coming from LoTW. I didn't look at the record before verification but if I enter a manual QSO now, the lookup from HAMQTH comes up with blanks for the State, not NSW.

Still, once this is corrected a re-download of the affected dates would put everything back in order and getting the state codes trimmed back to 2 characters for now should take care of the problems we're seeing. County names will still be wrong for now. That doesn't matter to me but maybe it would to others.
73 - Tom NS8K
NA5WH
Novice Class
Posts: 6
Joined: 14 Dec 2017, 05:13

Re: LoTW weirdness

Post by NA5WH »

G4POP wrote: 14 Dec 2017, 08:54 I have to ask why LOTW/ARRL keep going outside the standard ADIF format and thereby create issues for the users and developers?
I do wonder sometimes myself. ARRL wants to be the defacto standard, but they do seem to make it as hard as possible to use their service from time to time. Im not sure why it takes any time at all to add a new mode to the matching madness.

but yeah, the point of standardized data sets is to be standardized... current revision that I am aware of of ADIF:
"Enumeration for Country Code 291 (United States)" gives defined names for the states in 2 letter combinations. Japan's prefectures are 2 digit numbers, etc.

*sighs* Else create a new .ARRLDF or something.. :)

Ah well, such is life... but I would be really curious what developer thought "oh hey, maybe it would be great to include human readable text!".... and why...
NS8K
Advanced Class
Posts: 61
Joined: 29 Oct 2017, 12:54

Re: LoTW weirdness

Post by NS8K »

NS8K wrote: 14 Dec 2017, 22:08
w9mdb wrote: 14 Dec 2017, 16:34 If you download the sqlite3 command line utility you can fix this util LOTW or Log4OM fixes this.
Just run 'sqlite3 database.sqlite' with whatever your database name is and do this:
update log set state=SUBSTR(state,1,2);

de Mike W9MDB
Mike,

That was easy but yesterday I had a QSO with VK2DX in New South Wales. The state field is NSW // New South Wales which would become NS. Of course, the country would still be Australia. I think the NSW is coming from LoTW. I didn't look at the record before verification but if I enter a manual QSO now, the lookup from HAMQTH comes up with blanks for the State, not NSW.

Still, once this is corrected a re-download of the affected dates would put everything back in order and getting the state codes trimmed back to 2 characters for now should take care of the problems we're seeing. County names will still be wrong for now. That doesn't matter to me but maybe it would to others.
Mike,

I've thought about this a little more and I modified your SQL statement to this:

update log set state=SUBSTR(state,1,2) WHERE SUBSTR(state,4,2)="//";

Of course, this will update state only when positions 4&5 are //. That leaves my NSW record untouched. I could run another statement just for my 3 digit state codes or repair them manually. I've done this and now JTAlert is happy as is the awards section of Log4OM (until another LoTW download that is)!

If others are going to try this to repair their log it goes without saying - please backup before starting.
73 - Tom NS8K
Locked