[riemann-users] Protocol message end-group tag did not match expected tag

[Tahir Rauf]

I am running Riemann server 0.2.10 on 127.0.0.1:15555 . I have rsyslog conf
file which dictates to forward all messages matching particular pattern to
my Riemann server

$template ls_json,"{%timestamp:::date-rfc3339,jsonf:@timestamp%,%source:::jsonf:@source_host%,%msg:::json%}" :syslogtag,isequal,"tahir:" @127.0.0.1:15555;ls_json

Then I send a message using logger utility as following

logger "image [7a3cbb43:42:9]<340:01:52:53.032833>[50]: imghandler: Invoked for image" --tag="tahir"

I am expecting that Riemann will receive and print this message. However, I
got following error on Riemann side

WARN [2015-12-06 04:18:54,453] defaultEventExecutorGroup-2-1 - riemann.transport.udp - UDP handler caught
io.netty.handler.codec.DecoderException: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
at riemann.transport.proxy$io.netty.handler.codec.MessageToMessageDecoder$ff19274a.channelRead(Unknown Source)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
at io.netty.channel.nio.AbstractNioMessageChannel$NioMessageUnsafe.read(AbstractNioMessageChannel.java:93)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
at java.lang.Thread.run(Thread.java:745)Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.
at com.google.protobuf.InvalidProtocolBufferException.invalidEndTag(InvalidProtocolBufferException.java:94)
at com.google.protobuf.CodedInputStream.checkLastTagWas(CodedInputStream.java:174)
at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:139)
at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:168)
at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:174)
at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49)
at io.netty.handler.codec.protobuf.ProtobufDecoder.decode(ProtobufDecoder.java:119)
at io.netty.handler.codec.protobuf.ProtobufDecoder.decode(ProtobufDecoder.java:63)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89)
... 15 more

Can anyone tell me what I am doing wrong ???

[Aphyr]

I am running Riemann server 0.2.10 on 127.0.0.1:15555 . I have rsyslog conf file
which dictates to forward all messages matching particular pattern to my Riemann
server
|WARN [2015-12-0604:18:54,453]defaultEventExecutorGroup-2-1-riemann.transport.udp
-UDP handler caught
Can anyone tell me what I am doing wrong ???

UDP messages can be arbitrarily truncated. Why not use TCP?

--Kyle

[Tahir Rauf]

Thank you very much for your prompt response.

I agree that UDP message can be truncated arbitrarily and I will try TCP.
However, It seems that there is something else going on here as I am
receiving these warnings for EVERY event. There is a low probability that
EVERY message gets truncated. Is there any bug or am I doing something
obvious wrong?

Thanks again for your very prompt response.

Post by Tahir Rauf

I am running Riemann server 0.2.10 on 127.0.0.1:15555 . I have rsyslog conf file
which dictates to forward all messages matching particular pattern to my Riemann
server
|WARN [2015-12-0604
:18:54,453]defaultEventExecutorGroup-2-1-riemann.transport.udp
-UDP handler caught

Can anyone tell me what I am doing wrong ???
UDP messages can be arbitrarily truncated. Why not use TCP?
--Kyle
--
You received this message because you are subscribed to a topic in the
Google Groups "Riemann Users" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/riemann-users/_JpAXpHU8Pk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
For more options, visit https://groups.google.com/d/optout.

--
Tahir Rauf

--
[phone]

[Aphyr]

<p dir="ltr">Dunno! Your encoder could be broken, but 9/10 times this is the network truncating packets.</p> <div class="gmail_quote">On Dec 6, 2015 1:39 PM, Tahir Rauf &lt;***@gmail.com&gt; wrote:<br type='attribution'><blockquote class="quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Thank you very much for your prompt response.<div><br /><div>I agree that UDP message can be truncated arbitrarily and I will try TCP.</div><div>However, It seems that there is something else going on here as I am receiving these warnings for EVERY event. There is a low probability that EVERY message gets truncated. Is there any bug or am I doing something obvious wrong?</div><div><br /></div><div>Thanks again for your very prompt response.</div></div></div><div><br /><div class="elided-text">On Sun, Dec 6, 2015 at 12:20 PM, Aphyr <span dir="ltr">&lt;<a href="mailto:aphyr&#64;aphyr.com">aphyr&#64;aphyr.com</a>&gt;</span> wrote:<br /><blockquote style="margin:0 0 0 0.8ex;border-left:1px #ccc solid;padding-left:1ex">On 12/06/2015 12:16 PM, Tahir Rauf wrote:<br />
<blockquote style="margin:0 0 0 0.8ex;border-left:1px #ccc solid;padding-left:1ex">
<br />
I am running Riemann server 0.2.10 on <a href="http://127.0.0.1:15555">127.0.0.1:15555</a> . I have rsyslog conf file<br />
which dictates to forward all messages matching particular pattern to my Riemann<br />
server<br />
|WARN [2015-12-0604:18:54,453]defaultEventExecutorGroup-2-1-riemann.transport.udp<br />
-UDP handler caught<br />
</blockquote>
<br />
<blockquote style="margin:0 0 0 0.8ex;border-left:1px #ccc solid;padding-left:1ex">
Can anyone tell me what I am doing wrong ???<br />
</blockquote>
<br />
UDP messages can be arbitrarily truncated. Why not use TCP?<br />
<br />
--Kyle<div><div><br />
<br />
-- <br />
You received this message because you are subscribed to a topic in the Google Groups &#34;Riemann Users&#34; group.<br />
To unsubscribe from this topic, visit <a href="https://groups.google.com/d/topic/riemann-users/_JpAXpHU8Pk/unsubscribe">https://groups.google.com/d/topic/riemann-users/_JpAXpHU8Pk/unsubscribe</a>.<br />
To unsubscribe from this group and all its topics, send an email to <a href="mailto:riemann-users%2Bunsubscribe&#64;googlegroups.com">riemann-users&#43;unsubscribe&#64;googlegroups.com</a>.<br />
For more options, visit <a href="https://groups.google.com/d/optout">https://groups.google.com/d/optout</a>.<br />
</div></div></blockquote></div><br /><br clear="all" /><div><br /></div>-- <br /><div><div dir="ltr"><div>Tahir Rauf<br /></div><div><br /></div><div>--</div><div>[phone]</div><div><br /></div></div></div>
</div>

<p></p>

-- <br />
You received this message because you are subscribed to the Google Groups &#34;Riemann Users&#34; group.<br />
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="mailto:riemann-users&#43;unsubscribe&#64;googlegroups.com">riemann-users&#43;unsubscribe&#64;googlegroups.com</a>.<br />
For more options, visit <a href="https://groups.google.com/d/optout">https://groups.google.com/d/optout</a>.<br />
</blockquote></div>

<p></p>

-- <br />
You received this message because you are subscribed to the Google Groups &quot;Riemann Users&quot; group.<br />
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="mailto:riemann-users+***@googlegroups.com">riemann-users+***@googlegroups.com</a>.<br />
For more options, visit <a href="https://groups.google.com/d/optout">https://groups.google.com/d/optout</a>.<br />

[Marc Fournier]

Post by Tahir Rauf
I am running Riemann server 0.2.10 on 127.0.0.1:15555 . I have rsyslog conf
file which dictates to forward all messages matching particular pattern to
my Riemann server

With this you are sending JSON directly to Riemann's socket, aren't you ?

Riemann doesn't decode JSON-encoded events, only its own protobuf-based
protocol. And rsyslog doesn't have such an encoder.

You'll have to use something else than rsyslog, either as a replacement
or as a proxy. syslog-ng or logstash both can decode json sent over a
tcp socket or /dev/log and emit native riemann protobuf events. The
former having a much more performant encoder, which would make a
difference if your log rate is high.


HTH,
Marc

[Tahir Rauf]

Hi Marc,

Thanks alot. That precisely describes the problem.

Regards

Post by Marc Fournier

Post by Tahir Rauf
I am running Riemann server 0.2.10 on 127.0.0.1:15555 . I have rsyslog

conf

Post by Tahir Rauf
file which dictates to forward all messages matching particular pattern

to

Post by Tahir Rauf
my Riemann server

With this you are sending JSON directly to Riemann's socket, aren't you ?
Riemann doesn't decode JSON-encoded events, only its own protobuf-based
protocol. And rsyslog doesn't have such an encoder.
You'll have to use something else than rsyslog, either as a replacement
or as a proxy. syslog-ng or logstash both can decode json sent over a
tcp socket or /dev/log and emit native riemann protobuf events. The
former having a much more performant encoder, which would make a
difference if your log rate is high.
HTH,
Marc

--
Tahir Rauf

--
[phone]