Quantcast

HBase memstore log entries

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

HBase memstore log entries

aiyoh79
Hi,

http://pastebin.com/z9zPb49Y

Looking at the log entries above, is it normal to have different size for memsize (164.8k) in the first entry and Finished memstore flush of ~128.1m for the third entry?

My hbase version is 0.94.11.

aiyoh79
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: HBase memstore log entries

Jean-Marc Spaggiari
I will say yes. 128mb is the max size, but only the real content is flushed
and that's what is displayed. This value is memstoreSize.get();

JM


2013/9/26 aiyoh79 <[hidden email]>

> Hi,
>
> http://pastebin.com/z9zPb49Y
>
> Looking at the log entries above, is it normal to have different size for
> memsize (164.8k) in the first entry and Finished memstore flush of ~128.1m
> for the third entry?
>
> My hbase version is 0.94.11.
>
> aiyoh79
>
>
>
> --
> View this message in context:
> http://apache-hbase.679495.n3.nabble.com/HBase-memstore-log-entries-tp4051262.html
> Sent from the HBase User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: HBase memstore log entries

aiyoh79
So the actual data size for memstore is only 164.8k when flush is initiated right?

aiyoh79

Jean-Marc Spaggiari wrote
I will say yes. 128mb is the max size, but only the real content is flushed
and that's what is displayed. This value is memstoreSize.get();

JM


2013/9/26 aiyoh79 <[hidden email]>

> Hi,
>
> http://pastebin.com/z9zPb49Y
>
> Looking at the log entries above, is it normal to have different size for
> memsize (164.8k) in the first entry and Finished memstore flush of ~128.1m
> for the third entry?
>
> My hbase version is 0.94.11.
>
> aiyoh79
>
>
>
> --
> View this message in context:
> http://apache-hbase.679495.n3.nabble.com/HBase-memstore-log-entries-tp4051262.html
> Sent from the HBase User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: HBase memstore log entries

Jean-Marc Spaggiari
So, from the log to the source:

", memsize=" + StringUtils.humanReadableInt(flushed) +

And this variable is assigned hre:

flushed += this.memstore.heapSizeChange(kv, true);


Where heapSizeChange is:
  /*
   * Calculate how the MemStore size has changed.  Includes overhead of the
   * backing Map.
   * @param kv
   * @param notpresent True if the kv was NOT present in the set.
   * @return Size
   */
  long heapSizeChange(final KeyValue kv, final boolean notpresent)


So yes, the 160K are the part of the MemStore memory flushed in the
operation.

JM


2013/9/30 aiyoh79 <[hidden email]>

> So the actual data size for memstore is only 164.8k when flush is initiated
> right?
>
> aiyoh79
>
>
> Jean-Marc Spaggiari wrote
> > I will say yes. 128mb is the max size, but only the real content is
> > flushed
> > and that's what is displayed. This value is memstoreSize.get();
> >
> > JM
> >
> >
> > 2013/9/26 aiyoh79 &lt;
>
> > tchengwee@
>
> > &gt;
> >
> >> Hi,
> >>
> >> http://pastebin.com/z9zPb49Y
> >>
> >> Looking at the log entries above, is it normal to have different size
> for
> >> memsize (164.8k) in the first entry and Finished memstore flush of
> >> ~128.1m
> >> for the third entry?
> >>
> >> My hbase version is 0.94.11.
> >>
> >> aiyoh79
> >>
> >>
> >>
> >> --
> >> View this message in context:
> >>
> http://apache-hbase.679495.n3.nabble.com/HBase-memstore-log-entries-tp4051262.html
> >> Sent from the HBase User mailing list archive at Nabble.com.
> >>
>
>
>
>
>
> --
> View this message in context:
> http://apache-hbase.679495.n3.nabble.com/HBase-memstore-log-entries-tp4051262p4051341.html
> Sent from the HBase User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: HBase memstore log entries

aiyoh79
Thanks for the clarification. :)

aiyoh79

Jean-Marc Spaggiari wrote
So, from the log to the source:

", memsize=" + StringUtils.humanReadableInt(flushed) +

And this variable is assigned hre:

flushed += this.memstore.heapSizeChange(kv, true);


Where heapSizeChange is:
  /*
   * Calculate how the MemStore size has changed.  Includes overhead of the
   * backing Map.
   * @param kv
   * @param notpresent True if the kv was NOT present in the set.
   * @return Size
   */
  long heapSizeChange(final KeyValue kv, final boolean notpresent)


So yes, the 160K are the part of the MemStore memory flushed in the
operation.

JM


2013/9/30 aiyoh79 <[hidden email]>

> So the actual data size for memstore is only 164.8k when flush is initiated
> right?
>
> aiyoh79
>
>
> Jean-Marc Spaggiari wrote
> > I will say yes. 128mb is the max size, but only the real content is
> > flushed
> > and that's what is displayed. This value is memstoreSize.get();
> >
> > JM
> >
> >
> > 2013/9/26 aiyoh79 <
>
> > tchengwee@
>
> > >
> >
> >> Hi,
> >>
> >> http://pastebin.com/z9zPb49Y
> >>
> >> Looking at the log entries above, is it normal to have different size
> for
> >> memsize (164.8k) in the first entry and Finished memstore flush of
> >> ~128.1m
> >> for the third entry?
> >>
> >> My hbase version is 0.94.11.
> >>
> >> aiyoh79
> >>
> >>
> >>
> >> --
> >> View this message in context:
> >>
> http://apache-hbase.679495.n3.nabble.com/HBase-memstore-log-entries-tp4051262.html
> >> Sent from the HBase User mailing list archive at Nabble.com.
> >>
>
>
>
>
>
> --
> View this message in context:
> http://apache-hbase.679495.n3.nabble.com/HBase-memstore-log-entries-tp4051262p4051341.html
> Sent from the HBase User mailing list archive at Nabble.com.
>
Loading...