[Tinyos-help] multihop stuck after radio on/off?

George Lu georgedtlu at gmail.com
Thu Sep 22 14:37:32 PDT 2005


Hi there,

I have a test network of 3 sensing motes 205, 207, and 208 that are taking
sensor reading periodically upon receiving a new Drip message. The period is
currently set to 30 seconds for testing, but will increase to 5-10 minute
upon field deployment. After they finish taking sensor data, they would send
them to mote 0 attached to PC using multihopLQI. I also have a
Listen.javaon the same PC examining message traffic through a TOSBase
mote attached to
the same PC. All the motes are Tmote sky. My TinyOS version is 1.1.14.

The application ran well. To conserve energy, I added "call
RadioControl.stop();" a couple of seconds after the multihop message, and
added "call RadioControl.start();" a couple of seconds before the multihop
message. Only the three sensing motes would turn off the radio, since they
are the ones running on batteries. During test, a green LED on would
indicate a mote with radio on. So during every 30-second cycle I could see a
4-sec window where the green LED is on, and a brief red flash in the middle
of the 4-sec window indicating a multihop message sent from a sensing mote.

The application ran well initially. Then I started to observe one of the
sensing motes would be stuck with green LED on after some time, usually
close to an hour into the test. The stuck mote would not respond to nucleus
inquiries (mote 0 still does), neither could I reboot it with deluge.
Pressing the reset button would bring it back to life and in synch with the
other motes within a minute.

To be sure, I was able to run the same application with radio on all the
time flawlessly for a day. With the radio off for 26s in-between reporting
periods, at least one of the sensing motes would get stuck within the first
two hours. After 2-3 hours, usually only one of the three sensing motes is
left operating.

I searched the mailing list archives. I have moved all the "call Timer" into
a separate task outside of "timer.fired()" event handler. I have tried using
a much shorter BEACON PERIOD of 2 in MultiHopLQI.nc and have also added
"call RouteControl.manualUpdate();" as soon as radio is turned back on. I
have also increased the radio-on window from 4 seconds to 8 seconds. I have
not being able to solve the "stuck-at-green" problem.

I examined the message traffic captured by Listen.java on the TOSbase mote.
AM type 40 is my application's own multihop message from the three sensing
motes, 205 (0xCD), 207 (0xCF), 208 (0xD0) back to mote 0. AM type FA is by
multihopLQI. AM type 03 is the drip message to send commands to the sensing
motes.
 I noticed that at the moment of the final outgoing message from the stuck
mote, another mote would be sending more than one type 40 AM:


1A 21 08 5A FF FF 00 00 40 7D CD 00 CD 00 CC 02 CC 02 01 00 00 00 62 00 4F
03 D6 1C 00 00 00 00 00 00 5E 0F
1A 21 08 2E FF FF 00 00 40 7D CF 00 CF 00 C2 02 C2 02 01 00 00 00 62 00 EC
03 73 1B 00 00 00 00 00 00 7B 0F
1A 21 08 11 FF FF 00 00 40 7D D0 00 D0 00 97 02 97 02 01 00 00 00 62 00 68
03 95 1C 00 00 00 00 00 00 7B 0F
The above 3 lines expected behavior, a type 40 AM from each of CD, CF, D0.
0A 21 08 12 FF FF FF FF 03 7D 41 C8 63 00 23 00 00 68 00 00
14 21 08 13 FF FF FF FF FA 7D D0 00 D0 00 98 02 98 02 01 00 00 00 F3 00 01
00 00 00 00 00
14 21 08 5B FF FF FF FF FA 7D CD 00 CD 00 CD 02 CD 02 01 00 00 00 A5 00 01
00 00 00 00 00
0A 21 08 A0 FF FF FF FF 03 7D 41 C8 63 00 23 00 00 68 00 00
0E 21 08 A1 FF FF FF FF 03 7D 0B 02 00 00 00 00 00 00 00 00 00 00 00 00
14 21 08 2F FF FF FF FF FA 7D CF 00 CF 00 C3 02 C3 02 01 00 00 00 BD 00 01
00 00 00 00 00
Radio is turned off about now
14 21 08 A2 FF FF FF FF FA 7D 00 00 00 00 CC 05 CC 05 00 00 7E 00 00 00 00
00 00 00 00 00
0A 21 08 A3 FF FF FF FF 03 7D 41 C8 63 00 23 00 00 68 00 00
14 21 08 A4 FF FF FF FF FA 7D 00 00 00 00 CD 05 CD 05 00 00 7E 00 00 00 00
00 00 00 00 00
14 21 08 A5 FF FF FF FF FA 7D 00 00 00 00 CE 05 CE 05 00 00 7E 00 00 00 00
00 00 00 00 00
0A 21 08 A6 FF FF FF FF 03 7D 41 C8 63 00 23 00 00 68 00 00
14 21 08 A7 FF FF FF FF FA 7D 00 00 00 00 CF 05 CF 05 00 00 7E 00 00 00 00
00 00 00 00 00
14 21 08 A8 FF FF FF FF FA 7D 00 00 00 00 D0 05 D0 05 00 00 7E 00 00 00 00
00 00 00 00 00
0A 21 08 A9 FF FF FF FF 03 7D 41 C8 63 00 23 00 00 68 00 00
14 21 08 AA FF FF FF FF FA 7D 00 00 00 00 D1 05 D1 05 00 00 7E 00 00 00 00
00 00 00 00 00
14 21 08 AB FF FF FF FF FA 7D 00 00 00 00 D2 05 D2 05 00 00 7E 00 00 00 00
00 00 00 00 00
14 21 08 AC FF FF FF FF FA 7D 00 00 00 00 D3 05 D3 05 00 00 7E 00 00 00 00
00 00 00 00 00
14 21 08 AD FF FF FF FF FA 7D 00 00 00 00 D4 05 D4 05 00 00 7E 00 00 00 00
00 00 00 00 00
0A 21 08 AE FF FF FF FF 03 7D 41 C8 63 00 23 00 00 68 00 00
14 21 08 AF FF FF FF FF FA 7D 00 00 00 00 D5 05 D5 05 00 00 7E 00 00 00 00
00 00 00 00 00
14 21 08 B0 FF FF FF FF FA 7D 00 00 00 00 D6 05 D6 05 00 00 7E 00 00 00 00
00 00 00 00 00
14 21 08 B1 FF FF FF FF FA 7D 00 00 00 00 D7 05 D7 05 00 00 7E 00 00 00 00
00 00 00 00 00
14 21 08 B2 FF FF FF FF FA 7D 00 00 00 00 D8 05 D8 05 00 00 7E 00 00 00 00
00 00 00 00 00
14 21 08 B3 FF FF FF FF FA 7D 00 00 00 00 D9 05 D9 05 00 00 7E 00 00 00 00
00 00 00 00 00
Radio is turned on now, in the next few lines I see
several FA messages from sensing motes due to call RouteControl.manualUpdate
();
14 21 08 30 FF FF FF FF FA 7D CF 00 CF 00 C4 02 C4 02 01 00 00 00 BD 00 01
00 00 00 00 00
14 21 08 5C FF FF FF FF FA 7D CD 00 CD 00 CE 02 CE 02 01 00 00 00 A5 00 01
00 00 00 00 00
14 21 08 5D FF FF FF FF FA 7D CD 00 CD 00 CF 02 CF 02 01 00 00 00 A5 00 01
00 00 00 00 00
14 21 08 31 FF FF FF FF FA 7D CF 00 CF 00 C5 02 C5 02 01 00 00 00 BD 00 01
00 00 00 00 00
14 21 08 14 FF FF FF FF FA 7D D0 00 D0 00 99 02 99 02 01 00 00 00 A5 00 01
00 00 00 00 00
14 21 08 32 FF FF FF FF FA 7D CF 00 CF 00 C6 02 C6 02 01 00 00 00 BD 00 01
00 00 00 00 00
14 21 08 15 FF FF FF FF FA 7D D0 00 D0 00 9A 02 9A 02 01 00 00 00 A5 00 01
00 00 00 00 00
14 21 08 5F FF FF FF FF FA 7D CD 00 CD 00 D1 02 D1 02 01 00 00 00 A5 00 01
00 00 00 00 00
14 21 08 33 FF FF FF FF FA 7D CF 00 CF 00 C7 02 C7 02 01 00 00 00 BD 00 01
00 00 00 00 00
14 21 08 16 FF FF FF FF FA 7D D0 00 D0 00 9B 02 9B 02 01 00 00 00 A5 00 01
00 00 00 00 00
14 21 08 B4 FF FF FF FF FA 7D 00 00 00 00 DA 05 DA 05 00 00 7E 00 00 00 00
00 00 00 00 00
14 21 08 60 FF FF FF FF FA 7D CD 00 CD 00 D2 02 D2 02 01 00 00 00 A5 00 01
00 00 00 00 00
Drip message with command sequence number increased from 63 to 64,
triggering the 3 sensing motes into action
0A 21 08 B5 FF FF FF FF 03 7D 41 CA 64 00 23 00 00 68 00 00
0A 21 08 61 FF FF FF FF 03 7D 41 CA 64 00 23 00 00 68 00 00
14 21 08 34 FF FF FF FF FA 7D CF 00 CF 00 C8 02 C8 02 01 00 00 00 F3 00 01
00 00 00 00 00
0A 21 08 B6 FF FF FF FF 03 7D 41 CA 64 00 23 00 00 68 00 00
1A 21 08 62 FF FF 00 00 40 7D CD 00 CD 00 D3 02 D3 02 01 00 00 00 63 00 50
03 DC 1C 00 00 00 00 00 00 5F 0F
1A 21 08 35 FF FF 00 00 40 7D CF 00 CF 00 C9 02 C9 02 01 00 00 00 63 00 E8
03 76 1B 00 00 00 00 00 00 7B 0F
1A 21 08 17 FF FF 00 00 40 7D D0 00 D0 00 9C 02 9C 02 01 00 00 00 63 00 63
03 A3 1C 00 00 00 00 00 00 7B 0F
The above 3 lines looks good again. This is the last time they look normal.
14 21 08 36 FF FF FF FF FA 7D CF 00 CF 00 CA 02 CA 02 01 00 00 00 F3 00 01
00 00 00 00 00
14 21 08 63 FF FF FF FF FA 7D CD 00 CD 00 D4 02 D4 02 01 00 00 00 A5 00 01
00 00 00 00 00
Radio turned off about now.
0A 21 08 64 FF FF FF FF 03 7D 41 CA 64 00 23 00 00 68 00 00
0A 21 08 B7 FF FF FF FF 03 7D 41 CA 64 00 23 00 00 68 00 00
0A 21 08 37 FF FF FF FF 03 7D 41 CA 64 00 23 00 00 68 00 00
14 21 08 B8 FF FF FF FF FA 7D 00 00 00 00 DB 05 DB 05 00 00 7E 00 00 00 00
00 00 00 00 00
0A 21 08 B9 FF FF FF FF 03 7D 41 CA 64 00 23 00 00 68 00 00
14 21 08 BA FF FF FF FF FA 7D 00 00 00 00 DC 05 DC 05 00 00 7E 00 00 00 00
00 00 00 00 00
14 21 08 BB FF FF FF FF FA 7D 00 00 00 00 DD 05 DD 05 00 00 7E 00 00 00 00
00 00 00 00 00
0A 21 08 BC FF FF FF FF 03 7D 41 CA 64 00 23 00 00 68 00 00
14 21 08 BD FF FF FF FF FA 7D 00 00 00 00 DE 05 DE 05 00 00 7E 00 00 00 00
00 00 00 00 00
14 21 08 BE FF FF FF FF FA 7D 00 00 00 00 DF 05 DF 05 00 00 7E 00 00 00 00
00 00 00 00 00
0A 21 08 BF FF FF FF FF 03 7D 41 CA 64 00 23 00 00 68 00 00
14 21 08 C0 FF FF FF FF FA 7D 00 00 00 00 E0 05 E0 05 00 00 7E 00 00 00 00
00 00 00 00 00
14 21 08 C1 FF FF FF FF FA 7D 00 00 00 00 E1 05 E1 05 00 00 7E 00 00 00 00
00 00 00 00 00
14 21 08 C2 FF FF FF FF FA 7D 00 00 00 00 E2 05 E2 05 00 00 7E 00 00 00 00
00 00 00 00 00
14 21 08 C3 FF FF FF FF FA 7D 00 00 00 00 E3 05 E3 05 00 00 7E 00 00 00 00
00 00 00 00 00
0A 21 08 C4 FF FF FF FF 03 7D 41 CA 64 00 23 00 00 68 00 00
14 21 08 C5 FF FF FF FF FA 7D 00 00 00 00 E4 05 E4 05 00 00 7E 00 00 00 00
00 00 00 00 00
14 21 08 C6 FF FF FF FF FA 7D 00 00 00 00 E5 05 E5 05 00 00 7E 00 00 00 00
00 00 00 00 00
14 21 08 C7 FF FF FF FF FA 7D 00 00 00 00 E6 05 E6 05 00 00 7E 00 00 00 00
00 00 00 00 00
14 21 08 C8 FF FF FF FF FA 7D 00 00 00 00 E7 05 E7 05 00 00 7E 00 00 00 00
00 00 00 00 00
14 21 08 C9 FF FF FF FF FA 7D 00 00 00 00 E8 05 E8 05 00 00 7E 00 00 00 00
00 00 00 00 00
0A 21 08 19 FF FF FF FF 03 7D 41 CA 64 00 23 00 00 68 00 00
Radio turned on now. I noticed that there is no longer Type FA messages from
mote DO
14 21 08 65 FF FF FF FF FA 7D CD 00 CD 00 D5 02 D5 02 01 00 00 00 A5 00 01
00 00 00 00 00
14 21 08 38 FF FF FF FF FA 7D CF 00 CF 00 CB 02 CB 02 01 00 00 00 D8 00 01
00 00 00 00 00
14 21 08 66 FF FF FF FF FA 7D CD 00 CD 00 D6 02 D6 02 01 00 00 00 A5 00 01
00 00 00 00 00
14 21 08 39 FF FF FF FF FA 7D CF 00 CF 00 CC 02 CC 02 01 00 00 00 D8 00 01
00 00 00 00 00
14 21 08 67 FF FF FF FF FA 7D CD 00 CD 00 D7 02 D7 02 01 00 00 00 A5 00 01
00 00 00 00 00
14 21 08 68 FF FF FF FF FA 7D CD 00 CD 00 D8 02 D8 02 01 00 00 00 A5 00 01
00 00 00 00 00
14 21 08 3A FF FF FF FF FA 7D CF 00 CF 00 CD 02 CD 02 01 00 00 00 D8 00 01
00 00 00 00 00
14 21 08 CA FF FF FF FF FA 7D 00 00 00 00 E9 05 E9 05 00 00 7E 00 00 00 00
00 00 00 00 00
14 21 08 69 FF FF FF FF FA 7D CD 00 CD 00 D9 02 D9 02 01 00 00 00 A5 00 01
00 00 00 00 00
14 21 08 3B FF FF FF FF FA 7D CF 00 CF 00 CE 02 CE 02 01 00 00 00 D9 02 01
00 00 00 00 00
Drip message with command sequence number increased to 65.
0A 21 08 CB FF FF FF FF 03 7D 41 CC 65 00 23 00 00 68 00 00
0A 21 08 6A FF FF FF FF 03 7D 41 CC 65 00 23 00 00 68 00 00
0A 21 08 3C FF FF FF FF 03 7D 41 CC 65 00 23 00 00 68 00 00
14 21 08 3D FF FF FF FF FA 7D CF 00 CF 00 CF 02 CF 02 01 00 00 00 D9 02 01
00 00 00 00 00
0A 21 08 CC FF FF FF FF 03 7D 41 CC 65 00 23 00 00 68 00 00
1A 21 08 6B FF FF 00 00 40 7D CD 00 CD 00 DA 02 DA 02 01 00 00 00 64 00 4D
03 E5 1C 00 00 00 00 00 00 5B 0F
1A 21 08 3E FF FF 00 00 40 7D CF 00 CF 00 D0 02 D0 02 01 00 00 00 64 00 ED
03 76 1B 00 00 00 00 00 00 7E 0F
1A 21 08 3F FF FF 00 00 40 7D CF 00 CF 00 D0 02 D0 02 01 00 00 00 64 00 ED
03 76 1B 00 00 00 00 00 00 7E 0F
1A 21 08 1A FF FF 00 00 40 7D D0 00 D0 00 9F 02 9F 02 01 00 00 00 64 00 60
03 A9 1C 00 00 00 00 00 00 7B 0F
Above is the last communication from mote 208 (0xDO).
Notice that there were two type 40 AM from CF. Both have the same seq no of
64 in the payload but
0A 21 08 6C FF FF FF FF 03 7D 41 CC 65 00 23 00 00 68 00 00
0A 21 08 CD FF FF FF FF 03 7D 41 CC 65 00 23 00 00 68 00 00
14 21 08 6D FF FF FF FF FA 7D CD 00 CD 00 DB 02 DB 02 01 00 00 00 A5 00 01
00 00 00 00 00
14 21 08 CE FF FF FF FF FA 7D 00 00 00 00 EA 05 EA 05 00 00 7E 00 00 00 00
00 00 00 00 00
14 21 08 40 FF FF FF FF FA 7D CF 00 CF 00 D1 02 D1 02 01 00 00 00 D9 02 01
00 00 00 00 00
Radio turned off about now.
0A 21 08 CF FF FF FF FF 03 7D 41 CC 65 00 23 00 00 68 00 00
14 21 08 D0 FF FF FF FF FA 7D 00 00 00 00 EB 05 EB 05 00 00 7E 00 00 00 00
00 00 00 00 00
0A 21 08 D1 FF FF FF FF 03 7D 41 CC 65 00 23 00 00 68 00 00
14 21 08 D2 FF FF FF FF FA 7D 00 00 00 00 EC 05 EC 05 00 00 7E 00 00 00 00
00 00 00 00 00
14 21 08 D3 FF FF FF FF FA 7D 00 00 00 00 ED 05 ED 05 00 00 7E 00 00 00 00
00 00 00 00 00
14 21 08 D4 FF FF FF FF FA 7D 00 00 00 00 EE 05 EE 05 00 00 7E 00 00 00 00
00 00 00 00 00
14 21 08 D5 FF FF FF FF FA 7D 00 00 00 00 EF 05 EF 05 00 00 7E 00 00 00 00
00 00 00 00 00
0A 21 08 D6 FF FF FF FF 03 7D 41 CC 65 00 23 00 00 68 00 00
14 21 08 D7 FF FF FF FF FA 7D 00 00 00 00 F0 05 F0 05 00 00 7E 00 00 00 00
00 00 00 00 00
14 21 08 D8 FF FF FF FF FA 7D 00 00 00 00 F1 05 F1 05 00 00 7E 00 00 00 00
00 00 00 00 00
14 21 08 D9 FF FF FF FF FA 7D 00 00 00 00 F2 05 F2 05 00 00 7E 00 00 00 00
00 00 00 00 00
14 21 08 DA FF FF FF FF FA 7D 00 00 00 00 F3 05 F3 05 00 00 7E 00 00 00 00
00 00 00 00 00
14 21 08 DB FF FF FF FF FA 7D 00 00 00 00 F4 05 F4 05 00 00 7E 00 00 00 00
00 00 00 00 00
0A 21 08 DC FF FF FF FF 03 7D 41 CC 65 00 23 00 00 68 00 00
14 21 08 DD FF FF FF FF FA 7D 00 00 00 00 F5 05 F5 05 00 00 7E 00 00 00 00
00 00 00 00 00
14 21 08 DE FF FF FF FF FA 7D 00 00 00 00 F6 05 F6 05 00 00 7E 00 00 00 00
00 00 00 00 00
14 21 08 DF FF FF FF FF FA 7D 00 00 00 00 F7 05 F7 05 00 00 7E 00 00 00 00
00 00 00 00 00
Radio turned on.
14 21 08 6E FF FF FF FF FA 7D CD 00 CD 00 DC 02 DC 02 01 00 00 00 BD 00 01
00 00 00 00 00
14 21 08 41 FF FF FF FF FA 7D CF 00 CF 00 D2 02 D2 02 02 00 CD 00 CE 01 02
00 00 00 00 00
14 21 08 6F FF FF FF FF FA 7D CD 00 CD 00 DD 02 DD 02 01 00 00 00 BD 00 01
00 00 00 00 00
14 21 08 42 FF FF FF FF FA 7D CF 00 CF 00 D3 02 D3 02 02 00 CD 00 95 01 02
00 00 00 00 00
14 21 08 70 FF FF FF FF FA 7D CD 00 CD 00 DE 02 DE 02 01 00 00 00 BD 00 01
00 00 00 00 00
14 21 08 43 FF FF FF FF FA 7D CF 00 CF 00 D4 02 D4 02 02 00 CD 00 7A 01 02
00 00 00 00 00
14 21 08 E0 FF FF FF FF FA 7D 00 00 00 00 F8 05 F8 05 00 00 7E 00 00 00 00
00 00 00 00 00
14 21 08 71 FF FF FF FF FA 7D CD 00 CD 00 DF 02 DF 02 01 00 00 00 BD 00 01
00 00 00 00 00
Drip message with command sequence number increased to 66.
0A 21 08 E1 FF FF FF FF 03 7D 41 CE 66 00 23 00 00 68 00 00
0A 21 08 72 FF FF FF FF 03 7D 41 CE 66 00 23 00 00 68 00 00
14 21 08 73 FF FF FF FF FA 7D CD 00 CD 00 E0 02 E0 02 01 00 00 00 BD 00 01
00 00 00 00 00
0A 21 08 E2 FF FF FF FF 03 7D 41 CE 66 00 23 00 00 68 00 00
14 21 08 44 FF FF FF FF FA 7D CF 00 CF 00 D5 02 D5 02 01 00 00 00 D8 00 01
00 00 00 00 00
1A 21 08 74 FF FF 00 00 40 7D CD 00 CD 00 E1 02 E1 02 01 00 00 00 65 00 4E
03 E1 1C 00 00 00 00 00 00 4B 0F
1A 21 08 45 FF FF 00 00 40 7D CF 00 CF 00 D6 02 D6 02 01 00 00 00 65 00 E3
03 8C 1B 00 00 00 00 00 00 7E 0F
Mote D0 no longer reports in, CD and CF continue until one of them also gets
stuck

I can't say having multiple type 40 AM from another mote is the smoking gun
though, since there are also times I observed the stuck-at-green problem
without seeing multiple type 40 AM from sensing motes.
 Would really appreciate any advice/pointer to help resolve this issue.
 cheers,
George
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.millennium.berkeley.edu/pipermail/tinyos-help/attachments/20050922/2c232d5b/attachment-0001.html


More information about the Tinyos-help mailing list