Quantcast

keyvalue cache

classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

keyvalue cache

Enis Söztutar-2
Hi,

Before opening the issue, I though I should ask around first. What do you
think about a keyvalue cache sitting on top of the block cache? It is
mentioned in the big table paper, and it seems that zipfian kv access
patterns might benefit from something like this a lot. I could not find
anybody who proposed that before.

What do you guys think? Should we pursue a kv query-cache. My gut feeling
says that especially for some workloads we might gain significant
performance improvements, but we cannot verify it, until we implement and
profile it, right?

Thanks,
Enis
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: keyvalue cache

Dhruba Borthakur
+1 to evaluating a proposal for the rowcache. It could be very beneficial
to certain applications. The challenge is to reduce the memory overhad per
cached data.

-dhruba


On Tue, Apr 3, 2012 at 3:34 PM, Enis Söztutar <[hidden email]> wrote:

> Hi,
>
> Before opening the issue, I though I should ask around first. What do you
> think about a keyvalue cache sitting on top of the block cache? It is
> mentioned in the big table paper, and it seems that zipfian kv access
> patterns might benefit from something like this a lot. I could not find
> anybody who proposed that before.
>
> What do you guys think? Should we pursue a kv query-cache. My gut feeling
> says that especially for some workloads we might gain significant
> performance improvements, but we cannot verify it, until we implement and
> profile it, right?
>
> Thanks,
> Enis
>



--
Subscribe to my posts at http://www.facebook.com/dhruba
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: keyvalue cache

Vladimir Rodionov
In reply to this post by Enis Söztutar-2
Usually make sense for tables with random mostly access (point queries) For short-long scans block cache is preferable.
Cassandra has it (Row cache) but as since they cache the whole row (which can be very large) in many cases
it has sub par performance. Make sense to make caching configurable: table can use key-value cache and do not use block cache
and vice verse.

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: [hidden email]

________________________________________
From: Enis Söztutar [[hidden email]]
Sent: Tuesday, April 03, 2012 3:34 PM
To: [hidden email]
Subject: keyvalue cache

Hi,

Before opening the issue, I though I should ask around first. What do you
think about a keyvalue cache sitting on top of the block cache? It is
mentioned in the big table paper, and it seems that zipfian kv access
patterns might benefit from something like this a lot. I could not find
anybody who proposed that before.

What do you guys think? Should we pursue a kv query-cache. My gut feeling
says that especially for some workloads we might gain significant
performance improvements, but we cannot verify it, until we implement and
profile it, right?

Thanks,
Enis

Confidentiality Notice:  The information contained in this message, including any attachments hereto, may be confidential and is intended to be read only by the individual or entity to whom this message is addressed. If the reader of this message is not the intended recipient or an agent or designee of the intended recipient, please note that any review, use, disclosure or distribution of this message or its attachments, in any form, is strictly prohibited.  If you have received this message in error, please immediately notify the sender and/or [hidden email] and delete or destroy any copy of this message and its attachments.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: keyvalue cache

Enis Söztutar-2
As you said, caching the entire row does not make much sense, given that
the families are by contract the access boundaries. But caching column
families might be a good trade of for dealing with the per-item overhead.

Also agreed on cache being configurable at the table or better cf level. I
think we can do something like enable_block_cache = true,
enable_kv_cache=false, per column family.

Enis

On Tue, Apr 3, 2012 at 11:03 PM, Vladimir Rodionov
<[hidden email]>wrote:

> Usually make sense for tables with random mostly access (point queries)
> For short-long scans block cache is preferable.
> Cassandra has it (Row cache) but as since they cache the whole row (which
> can be very large) in many cases
> it has sub par performance. Make sense to make caching configurable: table
> can use key-value cache and do not use block cache
> and vice verse.
>
> Best regards,
> Vladimir Rodionov
> Principal Platform Engineer
> Carrier IQ, www.carrieriq.com
> e-mail: [hidden email]
>
> ________________________________________
> From: Enis Söztutar [[hidden email]]
> Sent: Tuesday, April 03, 2012 3:34 PM
> To: [hidden email]
> Subject: keyvalue cache
>
> Hi,
>
> Before opening the issue, I though I should ask around first. What do you
> think about a keyvalue cache sitting on top of the block cache? It is
> mentioned in the big table paper, and it seems that zipfian kv access
> patterns might benefit from something like this a lot. I could not find
> anybody who proposed that before.
>
> What do you guys think? Should we pursue a kv query-cache. My gut feeling
> says that especially for some workloads we might gain significant
> performance improvements, but we cannot verify it, until we implement and
> profile it, right?
>
> Thanks,
> Enis
>
> Confidentiality Notice:  The information contained in this message,
> including any attachments hereto, may be confidential and is intended to be
> read only by the individual or entity to whom this message is addressed. If
> the reader of this message is not the intended recipient or an agent or
> designee of the intended recipient, please note that any review, use,
> disclosure or distribution of this message or its attachments, in any form,
> is strictly prohibited.  If you have received this message in error, please
> immediately notify the sender and/or [hidden email] and
> delete or destroy any copy of this message and its attachments.
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: keyvalue cache

Matt Corgan
I guess the benefit of the KV cache is that you are not holding entire 64K
blocks in memory when you only care about 200 bytes of them.  Would an
alternative be to set a small block size (2KB or less)?

The problems with small block sizes would be expensive block cache
management overhead and inefficient scanning IO due to lack of read-ahead.
 Maybe improving the cache management and read-ahead would be more general
improvements that don't add as much complexity?

I'm having a hard time envisioning how you would do invalidations on the KV
cache and how you would merge its entries into a scan, etc.  Would it
basically be a memstore in front of the memstore where KVs get individually
invalidated instead of bulk-flushed?  Would it be sorted or hashed?

Matt

On Wed, Apr 4, 2012 at 10:35 AM, Enis Söztutar <[hidden email]> wrote:

> As you said, caching the entire row does not make much sense, given that
> the families are by contract the access boundaries. But caching column
> families might be a good trade of for dealing with the per-item overhead.
>
> Also agreed on cache being configurable at the table or better cf level. I
> think we can do something like enable_block_cache = true,
> enable_kv_cache=false, per column family.
>
> Enis
>
> On Tue, Apr 3, 2012 at 11:03 PM, Vladimir Rodionov
> <[hidden email]>wrote:
>
> > Usually make sense for tables with random mostly access (point queries)
> > For short-long scans block cache is preferable.
> > Cassandra has it (Row cache) but as since they cache the whole row (which
> > can be very large) in many cases
> > it has sub par performance. Make sense to make caching configurable:
> table
> > can use key-value cache and do not use block cache
> > and vice verse.
> >
> > Best regards,
> > Vladimir Rodionov
> > Principal Platform Engineer
> > Carrier IQ, www.carrieriq.com
> > e-mail: [hidden email]
> >
> > ________________________________________
> > From: Enis Söztutar [[hidden email]]
> > Sent: Tuesday, April 03, 2012 3:34 PM
> > To: [hidden email]
> > Subject: keyvalue cache
> >
> > Hi,
> >
> > Before opening the issue, I though I should ask around first. What do you
> > think about a keyvalue cache sitting on top of the block cache? It is
> > mentioned in the big table paper, and it seems that zipfian kv access
> > patterns might benefit from something like this a lot. I could not find
> > anybody who proposed that before.
> >
> > What do you guys think? Should we pursue a kv query-cache. My gut feeling
> > says that especially for some workloads we might gain significant
> > performance improvements, but we cannot verify it, until we implement and
> > profile it, right?
> >
> > Thanks,
> > Enis
> >
> > Confidentiality Notice:  The information contained in this message,
> > including any attachments hereto, may be confidential and is intended to
> be
> > read only by the individual or entity to whom this message is addressed.
> If
> > the reader of this message is not the intended recipient or an agent or
> > designee of the intended recipient, please note that any review, use,
> > disclosure or distribution of this message or its attachments, in any
> form,
> > is strictly prohibited.  If you have received this message in error,
> please
> > immediately notify the sender and/or [hidden email] and
> > delete or destroy any copy of this message and its attachments.
> >
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: keyvalue cache

stack-3
On Wed, Apr 4, 2012 at 11:40 AM, Matt Corgan <[hidden email]> wrote:
> I guess the benefit of the KV cache is that you are not holding entire 64K
> blocks in memory when you only care about 200 bytes of them.  Would an
> alternative be to set a small block size (2KB or less)?
>
> The problems with small block sizes would be expensive block cache
> management overhead and inefficient scanning IO due to lack of read-ahead.
>  Maybe improving the cache management and read-ahead would be more general
> improvements that don't add as much complexity?
>

I tend to think that there would be bigger bang for the buck doing
such as Matt describes above (plus things like the Todd started
MemStore improvements).

> I'm having a hard time envisioning how you would do invalidations on the KV
> cache and how you would merge its entries into a scan, etc.  Would it
> basically be a memstore in front of the memstore where KVs get individually
> invalidated instead of bulk-flushed?  Would it be sorted or hashed?
>

In the distant past, ruminations on a KV cache had it that it'd be
hard to do.  I could see it being good for hot cells or short, hot
rows -- it could make some some sub-ms savings I'd guess w/ some
savings in cpu -- for sure but it couldn't really be used by scanners
(least not w/o some interesting gymnastics).

St.Ack
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: keyvalue cache

Vladimir Rodionov
In reply to this post by Matt Corgan
1. 2KB can be too large for some applications. For example, some of our k-v sizes < 100 bytes combined.
2. These tables (from 1.) do not benefit from block cache at all (we did not try 100 B block size yet :)
3. And Matt is absolutely right: small block size is expensive

How about doing point queries on K-V cache and  bypass K-V cache on all Scans (when someone really need this)?
Implement K-V cache as a coprocessor application?

Invalidation of K-V entry is not necessary if all upserts operations go through K-V cache firstly if it sits in front of MemStore.
There will be no "stale or invalid" data situation in this case. Correct? No need for data to be sorted and no need for data to be merged
into a scan (we do not use K-V cache for Scans)


Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: [hidden email]

________________________________________
From: Matt Corgan [[hidden email]]
Sent: Wednesday, April 04, 2012 11:40 AM
To: [hidden email]
Subject: Re: keyvalue cache

I guess the benefit of the KV cache is that you are not holding entire 64K
blocks in memory when you only care about 200 bytes of them.  Would an
alternative be to set a small block size (2KB or less)?

The problems with small block sizes would be expensive block cache
management overhead and inefficient scanning IO due to lack of read-ahead.
 Maybe improving the cache management and read-ahead would be more general
improvements that don't add as much complexity?

I'm having a hard time envisioning how you would do invalidations on the KV
cache and how you would merge its entries into a scan, etc.  Would it
basically be a memstore in front of the memstore where KVs get individually
invalidated instead of bulk-flushed?  Would it be sorted or hashed?

Matt

On Wed, Apr 4, 2012 at 10:35 AM, Enis Söztutar <[hidden email]> wrote:

> As you said, caching the entire row does not make much sense, given that
> the families are by contract the access boundaries. But caching column
> families might be a good trade of for dealing with the per-item overhead.
>
> Also agreed on cache being configurable at the table or better cf level. I
> think we can do something like enable_block_cache = true,
> enable_kv_cache=false, per column family.
>
> Enis
>
> On Tue, Apr 3, 2012 at 11:03 PM, Vladimir Rodionov
> <[hidden email]>wrote:
>
> > Usually make sense for tables with random mostly access (point queries)
> > For short-long scans block cache is preferable.
> > Cassandra has it (Row cache) but as since they cache the whole row (which
> > can be very large) in many cases
> > it has sub par performance. Make sense to make caching configurable:
> table
> > can use key-value cache and do not use block cache
> > and vice verse.
> >
> > Best regards,
> > Vladimir Rodionov
> > Principal Platform Engineer
> > Carrier IQ, www.carrieriq.com
> > e-mail: [hidden email]
> >
> > ________________________________________
> > From: Enis Söztutar [[hidden email]]
> > Sent: Tuesday, April 03, 2012 3:34 PM
> > To: [hidden email]
> > Subject: keyvalue cache
> >
> > Hi,
> >
> > Before opening the issue, I though I should ask around first. What do you
> > think about a keyvalue cache sitting on top of the block cache? It is
> > mentioned in the big table paper, and it seems that zipfian kv access
> > patterns might benefit from something like this a lot. I could not find
> > anybody who proposed that before.
> >
> > What do you guys think? Should we pursue a kv query-cache. My gut feeling
> > says that especially for some workloads we might gain significant
> > performance improvements, but we cannot verify it, until we implement and
> > profile it, right?
> >
> > Thanks,
> > Enis
> >
> > Confidentiality Notice:  The information contained in this message,
> > including any attachments hereto, may be confidential and is intended to
> be
> > read only by the individual or entity to whom this message is addressed.
> If
> > the reader of this message is not the intended recipient or an agent or
> > designee of the intended recipient, please note that any review, use,
> > disclosure or distribution of this message or its attachments, in any
> form,
> > is strictly prohibited.  If you have received this message in error,
> please
> > immediately notify the sender and/or [hidden email] and
> > delete or destroy any copy of this message and its attachments.
> >
>

Confidentiality Notice:  The information contained in this message, including any attachments hereto, may be confidential and is intended to be read only by the individual or entity to whom this message is addressed. If the reader of this message is not the intended recipient or an agent or designee of the intended recipient, please note that any review, use, disclosure or distribution of this message or its attachments, in any form, is strictly prohibited.  If you have received this message in error, please immediately notify the sender and/or [hidden email] and delete or destroy any copy of this message and its attachments.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: keyvalue cache

Matt Corgan
It could act like a HashSet of KeyValues keyed on the
rowKey+family+qualifier but not including the timestamp.  As writes come in
it would evict or overwrite previous versions (read-through vs
write-through).  It would only service point queries where the
row+fam+qualifier are specified, returning the latest version.  Wouldn't be
able to do a typical rowKey-only Get (scan behind the scenes) because it
wouldn't know if it contained all the cells in the row, but if you could
specify all your row's qualifiers up-front it could work.


On Wed, Apr 4, 2012 at 2:30 PM, Vladimir Rodionov
<[hidden email]>wrote:

> 1. 2KB can be too large for some applications. For example, some of our
> k-v sizes < 100 bytes combined.
> 2. These tables (from 1.) do not benefit from block cache at all (we did
> not try 100 B block size yet :)
> 3. And Matt is absolutely right: small block size is expensive
>
> How about doing point queries on K-V cache and  bypass K-V cache on all
> Scans (when someone really need this)?
> Implement K-V cache as a coprocessor application?
>
> Invalidation of K-V entry is not necessary if all upserts operations go
> through K-V cache firstly if it sits in front of MemStore.
> There will be no "stale or invalid" data situation in this case. Correct?
> No need for data to be sorted and no need for data to be merged
> into a scan (we do not use K-V cache for Scans)
>
>
> Best regards,
> Vladimir Rodionov
> Principal Platform Engineer
> Carrier IQ, www.carrieriq.com
> e-mail: [hidden email]
>
> ________________________________________
> From: Matt Corgan [[hidden email]]
> Sent: Wednesday, April 04, 2012 11:40 AM
> To: [hidden email]
> Subject: Re: keyvalue cache
>
> I guess the benefit of the KV cache is that you are not holding entire 64K
> blocks in memory when you only care about 200 bytes of them.  Would an
> alternative be to set a small block size (2KB or less)?
>
> The problems with small block sizes would be expensive block cache
> management overhead and inefficient scanning IO due to lack of read-ahead.
>  Maybe improving the cache management and read-ahead would be more general
> improvements that don't add as much complexity?
>
> I'm having a hard time envisioning how you would do invalidations on the KV
> cache and how you would merge its entries into a scan, etc.  Would it
> basically be a memstore in front of the memstore where KVs get individually
> invalidated instead of bulk-flushed?  Would it be sorted or hashed?
>
> Matt
>
> On Wed, Apr 4, 2012 at 10:35 AM, Enis Söztutar <[hidden email]> wrote:
>
> > As you said, caching the entire row does not make much sense, given that
> > the families are by contract the access boundaries. But caching column
> > families might be a good trade of for dealing with the per-item overhead.
> >
> > Also agreed on cache being configurable at the table or better cf level.
> I
> > think we can do something like enable_block_cache = true,
> > enable_kv_cache=false, per column family.
> >
> > Enis
> >
> > On Tue, Apr 3, 2012 at 11:03 PM, Vladimir Rodionov
> > <[hidden email]>wrote:
> >
> > > Usually make sense for tables with random mostly access (point queries)
> > > For short-long scans block cache is preferable.
> > > Cassandra has it (Row cache) but as since they cache the whole row
> (which
> > > can be very large) in many cases
> > > it has sub par performance. Make sense to make caching configurable:
> > table
> > > can use key-value cache and do not use block cache
> > > and vice verse.
> > >
> > > Best regards,
> > > Vladimir Rodionov
> > > Principal Platform Engineer
> > > Carrier IQ, www.carrieriq.com
> > > e-mail: [hidden email]
> > >
> > > ________________________________________
> > > From: Enis Söztutar [[hidden email]]
> > > Sent: Tuesday, April 03, 2012 3:34 PM
> > > To: [hidden email]
> > > Subject: keyvalue cache
> > >
> > > Hi,
> > >
> > > Before opening the issue, I though I should ask around first. What do
> you
> > > think about a keyvalue cache sitting on top of the block cache? It is
> > > mentioned in the big table paper, and it seems that zipfian kv access
> > > patterns might benefit from something like this a lot. I could not find
> > > anybody who proposed that before.
> > >
> > > What do you guys think? Should we pursue a kv query-cache. My gut
> feeling
> > > says that especially for some workloads we might gain significant
> > > performance improvements, but we cannot verify it, until we implement
> and
> > > profile it, right?
> > >
> > > Thanks,
> > > Enis
> > >
> > > Confidentiality Notice:  The information contained in this message,
> > > including any attachments hereto, may be confidential and is intended
> to
> > be
> > > read only by the individual or entity to whom this message is
> addressed.
> > If
> > > the reader of this message is not the intended recipient or an agent or
> > > designee of the intended recipient, please note that any review, use,
> > > disclosure or distribution of this message or its attachments, in any
> > form,
> > > is strictly prohibited.  If you have received this message in error,
> > please
> > > immediately notify the sender and/or [hidden email] and
> > > delete or destroy any copy of this message and its attachments.
> > >
> >
>
> Confidentiality Notice:  The information contained in this message,
> including any attachments hereto, may be confidential and is intended to be
> read only by the individual or entity to whom this message is addressed. If
> the reader of this message is not the intended recipient or an agent or
> designee of the intended recipient, please note that any review, use,
> disclosure or distribution of this message or its attachments, in any form,
> is strictly prohibited.  If you have received this message in error, please
> immediately notify the sender and/or [hidden email] and
> delete or destroy any copy of this message and its attachments.
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: keyvalue cache

Matt Corgan
in the mean time, memcached could provide all those benefits without adding
any complexity to hbase...


On Wed, Apr 4, 2012 at 2:46 PM, Matt Corgan <[hidden email]> wrote:

> It could act like a HashSet of KeyValues keyed on the
> rowKey+family+qualifier but not including the timestamp.  As writes come in
> it would evict or overwrite previous versions (read-through vs
> write-through).  It would only service point queries where the
> row+fam+qualifier are specified, returning the latest version.  Wouldn't be
> able to do a typical rowKey-only Get (scan behind the scenes) because it
> wouldn't know if it contained all the cells in the row, but if you could
> specify all your row's qualifiers up-front it could work.
>
>
> On Wed, Apr 4, 2012 at 2:30 PM, Vladimir Rodionov <[hidden email]
> > wrote:
>
>> 1. 2KB can be too large for some applications. For example, some of our
>> k-v sizes < 100 bytes combined.
>> 2. These tables (from 1.) do not benefit from block cache at all (we did
>> not try 100 B block size yet :)
>> 3. And Matt is absolutely right: small block size is expensive
>>
>> How about doing point queries on K-V cache and  bypass K-V cache on all
>> Scans (when someone really need this)?
>> Implement K-V cache as a coprocessor application?
>>
>> Invalidation of K-V entry is not necessary if all upserts operations go
>> through K-V cache firstly if it sits in front of MemStore.
>> There will be no "stale or invalid" data situation in this case. Correct?
>> No need for data to be sorted and no need for data to be merged
>> into a scan (we do not use K-V cache for Scans)
>>
>>
>> Best regards,
>> Vladimir Rodionov
>> Principal Platform Engineer
>> Carrier IQ, www.carrieriq.com
>> e-mail: [hidden email]
>>
>> ________________________________________
>> From: Matt Corgan [[hidden email]]
>> Sent: Wednesday, April 04, 2012 11:40 AM
>> To: [hidden email]
>> Subject: Re: keyvalue cache
>>
>> I guess the benefit of the KV cache is that you are not holding entire 64K
>> blocks in memory when you only care about 200 bytes of them.  Would an
>> alternative be to set a small block size (2KB or less)?
>>
>> The problems with small block sizes would be expensive block cache
>> management overhead and inefficient scanning IO due to lack of read-ahead.
>>  Maybe improving the cache management and read-ahead would be more general
>> improvements that don't add as much complexity?
>>
>> I'm having a hard time envisioning how you would do invalidations on the
>> KV
>> cache and how you would merge its entries into a scan, etc.  Would it
>> basically be a memstore in front of the memstore where KVs get
>> individually
>> invalidated instead of bulk-flushed?  Would it be sorted or hashed?
>>
>> Matt
>>
>> On Wed, Apr 4, 2012 at 10:35 AM, Enis Söztutar <[hidden email]> wrote:
>>
>> > As you said, caching the entire row does not make much sense, given that
>> > the families are by contract the access boundaries. But caching column
>> > families might be a good trade of for dealing with the per-item
>> overhead.
>> >
>> > Also agreed on cache being configurable at the table or better cf
>> level. I
>> > think we can do something like enable_block_cache = true,
>> > enable_kv_cache=false, per column family.
>> >
>> > Enis
>> >
>> > On Tue, Apr 3, 2012 at 11:03 PM, Vladimir Rodionov
>> > <[hidden email]>wrote:
>> >
>> > > Usually make sense for tables with random mostly access (point
>> queries)
>> > > For short-long scans block cache is preferable.
>> > > Cassandra has it (Row cache) but as since they cache the whole row
>> (which
>> > > can be very large) in many cases
>> > > it has sub par performance. Make sense to make caching configurable:
>> > table
>> > > can use key-value cache and do not use block cache
>> > > and vice verse.
>> > >
>> > > Best regards,
>> > > Vladimir Rodionov
>> > > Principal Platform Engineer
>> > > Carrier IQ, www.carrieriq.com
>> > > e-mail: [hidden email]
>> > >
>> > > ________________________________________
>> > > From: Enis Söztutar [[hidden email]]
>> > > Sent: Tuesday, April 03, 2012 3:34 PM
>> > > To: [hidden email]
>> > > Subject: keyvalue cache
>> > >
>> > > Hi,
>> > >
>> > > Before opening the issue, I though I should ask around first. What do
>> you
>> > > think about a keyvalue cache sitting on top of the block cache? It is
>> > > mentioned in the big table paper, and it seems that zipfian kv access
>> > > patterns might benefit from something like this a lot. I could not
>> find
>> > > anybody who proposed that before.
>> > >
>> > > What do you guys think? Should we pursue a kv query-cache. My gut
>> feeling
>> > > says that especially for some workloads we might gain significant
>> > > performance improvements, but we cannot verify it, until we implement
>> and
>> > > profile it, right?
>> > >
>> > > Thanks,
>> > > Enis
>> > >
>> > > Confidentiality Notice:  The information contained in this message,
>> > > including any attachments hereto, may be confidential and is intended
>> to
>> > be
>> > > read only by the individual or entity to whom this message is
>> addressed.
>> > If
>> > > the reader of this message is not the intended recipient or an agent
>> or
>> > > designee of the intended recipient, please note that any review, use,
>> > > disclosure or distribution of this message or its attachments, in any
>> > form,
>> > > is strictly prohibited.  If you have received this message in error,
>> > please
>> > > immediately notify the sender and/or [hidden email] and
>> > > delete or destroy any copy of this message and its attachments.
>> > >
>> >
>>
>> Confidentiality Notice:  The information contained in this message,
>> including any attachments hereto, may be confidential and is intended to be
>> read only by the individual or entity to whom this message is addressed. If
>> the reader of this message is not the intended recipient or an agent or
>> designee of the intended recipient, please note that any review, use,
>> disclosure or distribution of this message or its attachments, in any form,
>> is strictly prohibited.  If you have received this message in error, please
>> immediately notify the sender and/or [hidden email] and
>> delete or destroy any copy of this message and its attachments.
>>
>
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: keyvalue cache

Vladimir Rodionov
In reply to this post by Matt Corgan
Yes, something like this. In many use cases only the latest (last) version matters,
so - no cell iterators, of course.

get_by (row+cf+qualifier) -> last version of. All other types of queries should bypass K-V cache.

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: [hidden email]

________________________________________
From: Matt Corgan [[hidden email]]
Sent: Wednesday, April 04, 2012 2:46 PM
To: [hidden email]
Subject: Re: keyvalue cache

It could act like a HashSet of KeyValues keyed on the
rowKey+family+qualifier but not including the timestamp.  As writes come in
it would evict or overwrite previous versions (read-through vs
write-through).  It would only service point queries where the
row+fam+qualifier are specified, returning the latest version.  Wouldn't be
able to do a typical rowKey-only Get (scan behind the scenes) because it
wouldn't know if it contained all the cells in the row, but if you could
specify all your row's qualifiers up-front it could work.


On Wed, Apr 4, 2012 at 2:30 PM, Vladimir Rodionov
<[hidden email]>wrote:

> 1. 2KB can be too large for some applications. For example, some of our
> k-v sizes < 100 bytes combined.
> 2. These tables (from 1.) do not benefit from block cache at all (we did
> not try 100 B block size yet :)
> 3. And Matt is absolutely right: small block size is expensive
>
> How about doing point queries on K-V cache and  bypass K-V cache on all
> Scans (when someone really need this)?
> Implement K-V cache as a coprocessor application?
>
> Invalidation of K-V entry is not necessary if all upserts operations go
> through K-V cache firstly if it sits in front of MemStore.
> There will be no "stale or invalid" data situation in this case. Correct?
> No need for data to be sorted and no need for data to be merged
> into a scan (we do not use K-V cache for Scans)
>
>
> Best regards,
> Vladimir Rodionov
> Principal Platform Engineer
> Carrier IQ, www.carrieriq.com
> e-mail: [hidden email]
>
> ________________________________________
> From: Matt Corgan [[hidden email]]
> Sent: Wednesday, April 04, 2012 11:40 AM
> To: [hidden email]
> Subject: Re: keyvalue cache
>
> I guess the benefit of the KV cache is that you are not holding entire 64K
> blocks in memory when you only care about 200 bytes of them.  Would an
> alternative be to set a small block size (2KB or less)?
>
> The problems with small block sizes would be expensive block cache
> management overhead and inefficient scanning IO due to lack of read-ahead.
>  Maybe improving the cache management and read-ahead would be more general
> improvements that don't add as much complexity?
>
> I'm having a hard time envisioning how you would do invalidations on the KV
> cache and how you would merge its entries into a scan, etc.  Would it
> basically be a memstore in front of the memstore where KVs get individually
> invalidated instead of bulk-flushed?  Would it be sorted or hashed?
>
> Matt
>
> On Wed, Apr 4, 2012 at 10:35 AM, Enis Söztutar <[hidden email]> wrote:
>
> > As you said, caching the entire row does not make much sense, given that
> > the families are by contract the access boundaries. But caching column
> > families might be a good trade of for dealing with the per-item overhead.
> >
> > Also agreed on cache being configurable at the table or better cf level.
> I
> > think we can do something like enable_block_cache = true,
> > enable_kv_cache=false, per column family.
> >
> > Enis
> >
> > On Tue, Apr 3, 2012 at 11:03 PM, Vladimir Rodionov
> > <[hidden email]>wrote:
> >
> > > Usually make sense for tables with random mostly access (point queries)
> > > For short-long scans block cache is preferable.
> > > Cassandra has it (Row cache) but as since they cache the whole row
> (which
> > > can be very large) in many cases
> > > it has sub par performance. Make sense to make caching configurable:
> > table
> > > can use key-value cache and do not use block cache
> > > and vice verse.
> > >
> > > Best regards,
> > > Vladimir Rodionov
> > > Principal Platform Engineer
> > > Carrier IQ, www.carrieriq.com
> > > e-mail: [hidden email]
> > >
> > > ________________________________________
> > > From: Enis Söztutar [[hidden email]]
> > > Sent: Tuesday, April 03, 2012 3:34 PM
> > > To: [hidden email]
> > > Subject: keyvalue cache
> > >
> > > Hi,
> > >
> > > Before opening the issue, I though I should ask around first. What do
> you
> > > think about a keyvalue cache sitting on top of the block cache? It is
> > > mentioned in the big table paper, and it seems that zipfian kv access
> > > patterns might benefit from something like this a lot. I could not find
> > > anybody who proposed that before.
> > >
> > > What do you guys think? Should we pursue a kv query-cache. My gut
> feeling
> > > says that especially for some workloads we might gain significant
> > > performance improvements, but we cannot verify it, until we implement
> and
> > > profile it, right?
> > >
> > > Thanks,
> > > Enis
> > >
> > > Confidentiality Notice:  The information contained in this message,
> > > including any attachments hereto, may be confidential and is intended
> to
> > be
> > > read only by the individual or entity to whom this message is
> addressed.
> > If
> > > the reader of this message is not the intended recipient or an agent or
> > > designee of the intended recipient, please note that any review, use,
> > > disclosure or distribution of this message or its attachments, in any
> > form,
> > > is strictly prohibited.  If you have received this message in error,
> > please
> > > immediately notify the sender and/or [hidden email] and
> > > delete or destroy any copy of this message and its attachments.
> > >
> >
>
> Confidentiality Notice:  The information contained in this message,
> including any attachments hereto, may be confidential and is intended to be
> read only by the individual or entity to whom this message is addressed. If
> the reader of this message is not the intended recipient or an agent or
> designee of the intended recipient, please note that any review, use,
> disclosure or distribution of this message or its attachments, in any form,
> is strictly prohibited.  If you have received this message in error, please
> immediately notify the sender and/or [hidden email] and
> delete or destroy any copy of this message and its attachments.
>

Confidentiality Notice:  The information contained in this message, including any attachments hereto, may be confidential and is intended to be read only by the individual or entity to whom this message is addressed. If the reader of this message is not the intended recipient or an agent or designee of the intended recipient, please note that any review, use, disclosure or distribution of this message or its attachments, in any form, is strictly prohibited.  If you have received this message in error, please immediately notify the sender and/or [hidden email] and delete or destroy any copy of this message and its attachments.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: keyvalue cache

Vladimir Rodionov
In reply to this post by Matt Corgan
"in the mean time, memcached could provide all those benefits without adding
any complexity to hbase..."

Tell this to Hypertable guys next time when they will be testing random gets performance :) .

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: [hidden email]






Confidentiality Notice:  The information contained in this message, including any attachments hereto, may be confidential and is intended to be read only by the individual or entity to whom this message is addressed. If the reader of this message is not the intended recipient or an agent or designee of the intended recipient, please note that any review, use, disclosure or distribution of this message or its attachments, in any form, is strictly prohibited.  If you have received this message in error, please immediately notify the sender and/or [hidden email] and delete or destroy any copy of this message and its attachments.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: keyvalue cache

Enis Söztutar-2
In reply to this post by Matt Corgan
Not sure about memcached or coprocessors based implementations, where you
would lose a consistent view over your data. I think one of the lucene over
hbase
implementation uses a memory cache (cant remember if it was memcache) over
hbase indexreaders and writers. You can do memcache deployments with 0 code
change to hbase, but haven't heard of any one other than those guys, no?
Has anyone
tried it?

On Wed, Apr 4, 2012 at 2:53 PM, Matt Corgan <[hidden email]> wrote:

> in the mean time, memcached could provide all those benefits without adding
> any complexity to hbase...
>
>
> On Wed, Apr 4, 2012 at 2:46 PM, Matt Corgan <[hidden email]> wrote:
>
> > It could act like a HashSet of KeyValues keyed on the
> > rowKey+family+qualifier but not including the timestamp.  As writes come
> in
> > it would evict or overwrite previous versions (read-through vs
> > write-through).  It would only service point queries where the
> > row+fam+qualifier are specified, returning the latest version.  Wouldn't
> be
> > able to do a typical rowKey-only Get (scan behind the scenes) because it
> > wouldn't know if it contained all the cells in the row, but if you could
> > specify all your row's qualifiers up-front it could work.
> >
> >
> > On Wed, Apr 4, 2012 at 2:30 PM, Vladimir Rodionov <
> [hidden email]
> > > wrote:
> >
> >> 1. 2KB can be too large for some applications. For example, some of our
> >> k-v sizes < 100 bytes combined.
> >> 2. These tables (from 1.) do not benefit from block cache at all (we did
> >> not try 100 B block size yet :)
> >> 3. And Matt is absolutely right: small block size is expensive
> >>
> >> How about doing point queries on K-V cache and  bypass K-V cache on all
> >> Scans (when someone really need this)?
> >> Implement K-V cache as a coprocessor application?
> >>
> >> Invalidation of K-V entry is not necessary if all upserts operations go
> >> through K-V cache firstly if it sits in front of MemStore.
> >> There will be no "stale or invalid" data situation in this case.
> Correct?
> >> No need for data to be sorted and no need for data to be merged
> >> into a scan (we do not use K-V cache for Scans)
> >>
> >>
> >> Best regards,
> >> Vladimir Rodionov
> >> Principal Platform Engineer
> >> Carrier IQ, www.carrieriq.com
> >> e-mail: [hidden email]
> >>
> >> ________________________________________
> >> From: Matt Corgan [[hidden email]]
> >> Sent: Wednesday, April 04, 2012 11:40 AM
> >> To: [hidden email]
> >> Subject: Re: keyvalue cache
> >>
> >> I guess the benefit of the KV cache is that you are not holding entire
> 64K
> >> blocks in memory when you only care about 200 bytes of them.  Would an
> >> alternative be to set a small block size (2KB or less)?
> >>
> >> The problems with small block sizes would be expensive block cache
> >> management overhead and inefficient scanning IO due to lack of
> read-ahead.
> >>  Maybe improving the cache management and read-ahead would be more
> general
> >> improvements that don't add as much complexity?
> >>
> >> I'm having a hard time envisioning how you would do invalidations on the
> >> KV
> >> cache and how you would merge its entries into a scan, etc.  Would it
> >> basically be a memstore in front of the memstore where KVs get
> >> individually
> >> invalidated instead of bulk-flushed?  Would it be sorted or hashed?
> >>
> >> Matt
> >>
> >> On Wed, Apr 4, 2012 at 10:35 AM, Enis Söztutar <[hidden email]> wrote:
> >>
> >> > As you said, caching the entire row does not make much sense, given
> that
> >> > the families are by contract the access boundaries. But caching column
> >> > families might be a good trade of for dealing with the per-item
> >> overhead.
> >> >
> >> > Also agreed on cache being configurable at the table or better cf
> >> level. I
> >> > think we can do something like enable_block_cache = true,
> >> > enable_kv_cache=false, per column family.
> >> >
> >> > Enis
> >> >
> >> > On Tue, Apr 3, 2012 at 11:03 PM, Vladimir Rodionov
> >> > <[hidden email]>wrote:
> >> >
> >> > > Usually make sense for tables with random mostly access (point
> >> queries)
> >> > > For short-long scans block cache is preferable.
> >> > > Cassandra has it (Row cache) but as since they cache the whole row
> >> (which
> >> > > can be very large) in many cases
> >> > > it has sub par performance. Make sense to make caching configurable:
> >> > table
> >> > > can use key-value cache and do not use block cache
> >> > > and vice verse.
> >> > >
> >> > > Best regards,
> >> > > Vladimir Rodionov
> >> > > Principal Platform Engineer
> >> > > Carrier IQ, www.carrieriq.com
> >> > > e-mail: [hidden email]
> >> > >
> >> > > ________________________________________
> >> > > From: Enis Söztutar [[hidden email]]
> >> > > Sent: Tuesday, April 03, 2012 3:34 PM
> >> > > To: [hidden email]
> >> > > Subject: keyvalue cache
> >> > >
> >> > > Hi,
> >> > >
> >> > > Before opening the issue, I though I should ask around first. What
> do
> >> you
> >> > > think about a keyvalue cache sitting on top of the block cache? It
> is
> >> > > mentioned in the big table paper, and it seems that zipfian kv
> access
> >> > > patterns might benefit from something like this a lot. I could not
> >> find
> >> > > anybody who proposed that before.
> >> > >
> >> > > What do you guys think? Should we pursue a kv query-cache. My gut
> >> feeling
> >> > > says that especially for some workloads we might gain significant
> >> > > performance improvements, but we cannot verify it, until we
> implement
> >> and
> >> > > profile it, right?
> >> > >
> >> > > Thanks,
> >> > > Enis
> >> > >
> >> > > Confidentiality Notice:  The information contained in this message,
> >> > > including any attachments hereto, may be confidential and is
> intended
> >> to
> >> > be
> >> > > read only by the individual or entity to whom this message is
> >> addressed.
> >> > If
> >> > > the reader of this message is not the intended recipient or an agent
> >> or
> >> > > designee of the intended recipient, please note that any review,
> use,
> >> > > disclosure or distribution of this message or its attachments, in
> any
> >> > form,
> >> > > is strictly prohibited.  If you have received this message in error,
> >> > please
> >> > > immediately notify the sender and/or [hidden email]
> >> > > delete or destroy any copy of this message and its attachments.
> >> > >
> >> >
> >>
> >> Confidentiality Notice:  The information contained in this message,
> >> including any attachments hereto, may be confidential and is intended
> to be
> >> read only by the individual or entity to whom this message is
> addressed. If
> >> the reader of this message is not the intended recipient or an agent or
> >> designee of the intended recipient, please note that any review, use,
> >> disclosure or distribution of this message or its attachments, in any
> form,
> >> is strictly prohibited.  If you have received this message in error,
> please
> >> immediately notify the sender and/or [hidden email] and
> >> delete or destroy any copy of this message and its attachments.
> >>
> >
> >
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: keyvalue cache

Andrew Purtell
In reply to this post by Matt Corgan
I thought about trying this out once with a coprocessor, hooking the Gets, with an embedded Hazelcast. That would just be a proof of concept. The idea is to scale the KV cache independent of regionserver limits (maybe we're only giving 1 GB per RS to the value cache and a 10 GB region is hot) and the next step could be modifying the client to spread read load over replicas (HBASE-2357). This doesn't consider scans either.


Best regards,

    - Andy


Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)



>________________________________
> From: Matt Corgan <[hidden email]>
>To: [hidden email]
>Sent: Wednesday, April 4, 2012 2:46 PM
>Subject: Re: keyvalue cache
>
>It could act like a HashSet of KeyValues keyed on the
>rowKey+family+qualifier but not including the timestamp.  As writes come in
>it would evict or overwrite previous versions (read-through vs
>write-through).  It would only service point queries where the
>row+fam+qualifier are specified, returning the latest version.  Wouldn't be
>able to do a typical rowKey-only Get (scan behind the scenes) because it
>wouldn't know if it contained all the cells in the row, but if you could
>specify all your row's qualifiers up-front it could work.
>
>
>On Wed, Apr 4, 2012 at 2:30 PM, Vladimir Rodionov
><[hidden email]>wrote:
>
>> 1. 2KB can be too large for some applications. For example, some of our
>> k-v sizes < 100 bytes combined.
>> 2. These tables (from 1.) do not benefit from block cache at all (we did
>> not try 100 B block size yet :)
>> 3. And Matt is absolutely right: small block size is expensive
>>
>> How about doing point queries on K-V cache and  bypass K-V cache on all
>> Scans (when someone really need this)?
>> Implement K-V cache as a coprocessor application?
>>
>> Invalidation of K-V entry is not necessary if all upserts operations go
>> through K-V cache firstly if it sits in front of MemStore.
>> There will be no "stale or invalid" data situation in this case. Correct?
>> No need for data to be sorted and no need for data to be merged
>> into a scan (we do not use K-V cache for Scans)
>>
>>
>> Best regards,
>> Vladimir Rodionov
>> Principal Platform Engineer
>> Carrier IQ, www.carrieriq.com
>> e-mail: [hidden email]
>>
>> ________________________________________
>> From: Matt Corgan [[hidden email]]
>> Sent: Wednesday, April 04, 2012 11:40 AM
>> To: [hidden email]
>> Subject: Re: keyvalue cache
>>
>> I guess the benefit of the KV cache is that you are not holding entire 64K
>> blocks in memory when you only care about 200 bytes of them.  Would an
>> alternative be to set a small block size (2KB or less)?
>>
>> The problems with small block sizes would be expensive block cache
>> management overhead and inefficient scanning IO due to lack of read-ahead.
>>  Maybe improving the cache management and read-ahead would be more general
>> improvements that don't add as much complexity?
>>
>> I'm having a hard time envisioning how you would do invalidations on the KV
>> cache and how you would merge its entries into a scan, etc.  Would it
>> basically be a memstore in front of the memstore where KVs get individually
>> invalidated instead of bulk-flushed?  Would it be sorted or hashed?
>>
>> Matt
>>
>> On Wed, Apr 4, 2012 at 10:35 AM, Enis Söztutar <[hidden email]> wrote:
>>
>> > As you said, caching the entire row does not make much sense, given that
>> > the families are by contract the access boundaries. But caching column
>> > families might be a good trade of for dealing with the per-item overhead.
>> >
>> > Also agreed on cache being configurable at the table or better cf level.
>> I
>> > think we can do something like enable_block_cache = true,
>> > enable_kv_cache=false, per column family.
>> >
>> > Enis
>> >
>> > On Tue, Apr 3, 2012 at 11:03 PM, Vladimir Rodionov
>> > <[hidden email]>wrote:
>> >
>> > > Usually make sense for tables with random mostly access (point queries)
>> > > For short-long scans block cache is preferable.
>> > > Cassandra has it (Row cache) but as since they cache the whole row
>> (which
>> > > can be very large) in many cases
>> > > it has sub par performance. Make sense to make caching configurable:
>> > table
>> > > can use key-value cache and do not use block cache
>> > > and vice verse.
>> > >
>> > > Best regards,
>> > > Vladimir Rodionov
>> > > Principal Platform Engineer
>> > > Carrier IQ, www.carrieriq.com
>> > > e-mail: [hidden email]
>> > >
>> > > ________________________________________
>> > > From: Enis Söztutar [[hidden email]]
>> > > Sent: Tuesday, April 03, 2012 3:34 PM
>> > > To: [hidden email]
>> > > Subject: keyvalue cache
>> > >
>> > > Hi,
>> > >
>> > > Before opening the issue, I though I should ask around first. What do
>> you
>> > > think about a keyvalue cache sitting on top of the block cache? It is
>> > > mentioned in the big table paper, and it seems that zipfian kv access
>> > > patterns might benefit from something like this a lot. I could not find
>> > > anybody who proposed that before.
>> > >
>> > > What do you guys think? Should we pursue a kv query-cache. My gut
>> feeling
>> > > says that especially for some workloads we might gain significant
>> > > performance improvements, but we cannot verify it, until we implement
>> and
>> > > profile it, right?
>> > >
>> > > Thanks,
>> > > Enis
>> > >
>> > > Confidentiality Notice:  The information contained in this message,
>> > > including any attachments hereto, may be confidential and is intended
>> to
>> > be
>> > > read only by the individual or entity to whom this message is
>> addressed.
>> > If
>> > > the reader of this message is not the intended recipient or an agent or
>> > > designee of the intended recipient, please note that any review, use,
>> > > disclosure or distribution of this message or its attachments, in any
>> > form,
>> > > is strictly prohibited.  If you have received this message in error,
>> > please
>> > > immediately notify the sender and/or [hidden email] and
>> > > delete or destroy any copy of this message and its attachments.
>> > >
>> >
>>
>> Confidentiality Notice:  The information contained in this message,
>> including any attachments hereto, may be confidential and is intended to be
>> read only by the individual or entity to whom this message is addressed. If
>> the reader of this message is not the intended recipient or an agent or
>> designee of the intended recipient, please note that any review, use,
>> disclosure or distribution of this message or its attachments, in any form,
>> is strictly prohibited.  If you have received this message in error, please
>> immediately notify the sender and/or [hidden email] and
>> delete or destroy any copy of this message and its attachments.
>>
>
>
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: keyvalue cache

Matt Corgan
A client-side memcached setup can stay pretty consistent if you send all of
your puts and deletes through it before sending them to hbase, but yeah, I
guess you lose strict consistency under heavy read/write from multiple
simultaneous clients.  But, like Andy is saying, if you route the requests
through the regionserver and it talks to memcached/hazelcast, couldn't that
be fully consistent?


On Wed, Apr 4, 2012 at 3:09 PM, Andrew Purtell <[hidden email]> wrote:

> I thought about trying this out once with a coprocessor, hooking the Gets,
> with an embedded Hazelcast. That would just be a proof of concept. The idea
> is to scale the KV cache independent of regionserver limits (maybe we're
> only giving 1 GB per RS to the value cache and a 10 GB region is hot) and
> the next step could be modifying the client to spread read load over
> replicas (HBASE-2357). This doesn't consider scans either.
>
>
> Best regards,
>
>     - Andy
>
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>
>
>
> >________________________________
> > From: Matt Corgan <[hidden email]>
> >To: [hidden email]
> >Sent: Wednesday, April 4, 2012 2:46 PM
> >Subject: Re: keyvalue cache
> >
> >It could act like a HashSet of KeyValues keyed on the
> >rowKey+family+qualifier but not including the timestamp.  As writes come
> in
> >it would evict or overwrite previous versions (read-through vs
> >write-through).  It would only service point queries where the
> >row+fam+qualifier are specified, returning the latest version.  Wouldn't
> be
> >able to do a typical rowKey-only Get (scan behind the scenes) because it
> >wouldn't know if it contained all the cells in the row, but if you could
> >specify all your row's qualifiers up-front it could work.
> >
> >
> >On Wed, Apr 4, 2012 at 2:30 PM, Vladimir Rodionov
> ><[hidden email]>wrote:
> >
> >> 1. 2KB can be too large for some applications. For example, some of our
> >> k-v sizes < 100 bytes combined.
> >> 2. These tables (from 1.) do not benefit from block cache at all (we did
> >> not try 100 B block size yet :)
> >> 3. And Matt is absolutely right: small block size is expensive
> >>
> >> How about doing point queries on K-V cache and  bypass K-V cache on all
> >> Scans (when someone really need this)?
> >> Implement K-V cache as a coprocessor application?
> >>
> >> Invalidation of K-V entry is not necessary if all upserts operations go
> >> through K-V cache firstly if it sits in front of MemStore.
> >> There will be no "stale or invalid" data situation in this case.
> Correct?
> >> No need for data to be sorted and no need for data to be merged
> >> into a scan (we do not use K-V cache for Scans)
> >>
> >>
> >> Best regards,
> >> Vladimir Rodionov
> >> Principal Platform Engineer
> >> Carrier IQ, www.carrieriq.com
> >> e-mail: [hidden email]
> >>
> >> ________________________________________
> >> From: Matt Corgan [[hidden email]]
> >> Sent: Wednesday, April 04, 2012 11:40 AM
> >> To: [hidden email]
> >> Subject: Re: keyvalue cache
> >>
> >> I guess the benefit of the KV cache is that you are not holding entire
> 64K
> >> blocks in memory when you only care about 200 bytes of them.  Would an
> >> alternative be to set a small block size (2KB or less)?
> >>
> >> The problems with small block sizes would be expensive block cache
> >> management overhead and inefficient scanning IO due to lack of
> read-ahead.
> >>  Maybe improving the cache management and read-ahead would be more
> general
> >> improvements that don't add as much complexity?
> >>
> >> I'm having a hard time envisioning how you would do invalidations on
> the KV
> >> cache and how you would merge its entries into a scan, etc.  Would it
> >> basically be a memstore in front of the memstore where KVs get
> individually
> >> invalidated instead of bulk-flushed?  Would it be sorted or hashed?
> >>
> >> Matt
> >>
> >> On Wed, Apr 4, 2012 at 10:35 AM, Enis Söztutar <[hidden email]> wrote:
> >>
> >> > As you said, caching the entire row does not make much sense, given
> that
> >> > the families are by contract the access boundaries. But caching column
> >> > families might be a good trade of for dealing with the per-item
> overhead.
> >> >
> >> > Also agreed on cache being configurable at the table or better cf
> level.
> >> I
> >> > think we can do something like enable_block_cache = true,
> >> > enable_kv_cache=false, per column family.
> >> >
> >> > Enis
> >> >
> >> > On Tue, Apr 3, 2012 at 11:03 PM, Vladimir Rodionov
> >> > <[hidden email]>wrote:
> >> >
> >> > > Usually make sense for tables with random mostly access (point
> queries)
> >> > > For short-long scans block cache is preferable.
> >> > > Cassandra has it (Row cache) but as since they cache the whole row
> >> (which
> >> > > can be very large) in many cases
> >> > > it has sub par performance. Make sense to make caching configurable:
> >> > table
> >> > > can use key-value cache and do not use block cache
> >> > > and vice verse.
> >> > >
> >> > > Best regards,
> >> > > Vladimir Rodionov
> >> > > Principal Platform Engineer
> >> > > Carrier IQ, www.carrieriq.com
> >> > > e-mail: [hidden email]
> >> > >
> >> > > ________________________________________
> >> > > From: Enis Söztutar [[hidden email]]
> >> > > Sent: Tuesday, April 03, 2012 3:34 PM
> >> > > To: [hidden email]
> >> > > Subject: keyvalue cache
> >> > >
> >> > > Hi,
> >> > >
> >> > > Before opening the issue, I though I should ask around first. What
> do
> >> you
> >> > > think about a keyvalue cache sitting on top of the block cache? It
> is
> >> > > mentioned in the big table paper, and it seems that zipfian kv
> access
> >> > > patterns might benefit from something like this a lot. I could not
> find
> >> > > anybody who proposed that before.
> >> > >
> >> > > What do you guys think? Should we pursue a kv query-cache. My gut
> >> feeling
> >> > > says that especially for some workloads we might gain significant
> >> > > performance improvements, but we cannot verify it, until we
> implement
> >> and
> >> > > profile it, right?
> >> > >
> >> > > Thanks,
> >> > > Enis
> >> > >
> >> > > Confidentiality Notice:  The information contained in this message,
> >> > > including any attachments hereto, may be confidential and is
> intended
> >> to
> >> > be
> >> > > read only by the individual or entity to whom this message is
> >> addressed.
> >> > If
> >> > > the reader of this message is not the intended recipient or an
> agent or
> >> > > designee of the intended recipient, please note that any review,
> use,
> >> > > disclosure or distribution of this message or its attachments, in
> any
> >> > form,
> >> > > is strictly prohibited.  If you have received this message in error,
> >> > please
> >> > > immediately notify the sender and/or [hidden email]
> >> > > delete or destroy any copy of this message and its attachments.
> >> > >
> >> >
> >>
> >> Confidentiality Notice:  The information contained in this message,
> >> including any attachments hereto, may be confidential and is intended
> to be
> >> read only by the individual or entity to whom this message is
> addressed. If
> >> the reader of this message is not the intended recipient or an agent or
> >> designee of the intended recipient, please note that any review, use,
> >> disclosure or distribution of this message or its attachments, in any
> form,
> >> is strictly prohibited.  If you have received this message in error,
> please
> >> immediately notify the sender and/or [hidden email] and
> >> delete or destroy any copy of this message and its attachments.
> >>
> >
> >
> >
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: keyvalue cache

Enis Söztutar-2
I think you are right that if you replicate the row MVCC semantics in the
cache, then
you can get a consistent view. I was referring to a more client side
approach.

I guess the take aways are:
 - forget about scans, and shoot for point gets, which I agree
 - per-kv cache overhead might be huge, but still worth trying it out.
 - can also be architected on top of coprocessors
 - might be complex to implement, but still some use cases would benefit a
lot.
 - row cache / family cache / kv cache

Thanks for all the input!

Enis

On Wed, Apr 4, 2012 at 3:28 PM, Matt Corgan <[hidden email]> wrote:

> A client-side memcached setup can stay pretty consistent if you send all of
> your puts and deletes through it before sending them to hbase, but yeah, I
> guess you lose strict consistency under heavy read/write from multiple
> simultaneous clients.  But, like Andy is saying, if you route the requests
> through the regionserver and it talks to memcached/hazelcast, couldn't that
> be fully consistent?
>
>
> On Wed, Apr 4, 2012 at 3:09 PM, Andrew Purtell <[hidden email]>
> wrote:
>
> > I thought about trying this out once with a coprocessor, hooking the
> Gets,
> > with an embedded Hazelcast. That would just be a proof of concept. The
> idea
> > is to scale the KV cache independent of regionserver limits (maybe we're
> > only giving 1 GB per RS to the value cache and a 10 GB region is hot) and
> > the next step could be modifying the client to spread read load over
> > replicas (HBASE-2357). This doesn't consider scans either.
> >
> >
> > Best regards,
> >
> >     - Andy
> >
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
> > (via Tom White)
> >
> >
> >
> > >________________________________
> > > From: Matt Corgan <[hidden email]>
> > >To: [hidden email]
> > >Sent: Wednesday, April 4, 2012 2:46 PM
> > >Subject: Re: keyvalue cache
> > >
> > >It could act like a HashSet of KeyValues keyed on the
> > >rowKey+family+qualifier but not including the timestamp.  As writes come
> > in
> > >it would evict or overwrite previous versions (read-through vs
> > >write-through).  It would only service point queries where the
> > >row+fam+qualifier are specified, returning the latest version.  Wouldn't
> > be
> > >able to do a typical rowKey-only Get (scan behind the scenes) because it
> > >wouldn't know if it contained all the cells in the row, but if you could
> > >specify all your row's qualifiers up-front it could work.
> > >
> > >
> > >On Wed, Apr 4, 2012 at 2:30 PM, Vladimir Rodionov
> > ><[hidden email]>wrote:
> > >
> > >> 1. 2KB can be too large for some applications. For example, some of
> our
> > >> k-v sizes < 100 bytes combined.
> > >> 2. These tables (from 1.) do not benefit from block cache at all (we
> did
> > >> not try 100 B block size yet :)
> > >> 3. And Matt is absolutely right: small block size is expensive
> > >>
> > >> How about doing point queries on K-V cache and  bypass K-V cache on
> all
> > >> Scans (when someone really need this)?
> > >> Implement K-V cache as a coprocessor application?
> > >>
> > >> Invalidation of K-V entry is not necessary if all upserts operations
> go
> > >> through K-V cache firstly if it sits in front of MemStore.
> > >> There will be no "stale or invalid" data situation in this case.
> > Correct?
> > >> No need for data to be sorted and no need for data to be merged
> > >> into a scan (we do not use K-V cache for Scans)
> > >>
> > >>
> > >> Best regards,
> > >> Vladimir Rodionov
> > >> Principal Platform Engineer
> > >> Carrier IQ, www.carrieriq.com
> > >> e-mail: [hidden email]
> > >>
> > >> ________________________________________
> > >> From: Matt Corgan [[hidden email]]
> > >> Sent: Wednesday, April 04, 2012 11:40 AM
> > >> To: [hidden email]
> > >> Subject: Re: keyvalue cache
> > >>
> > >> I guess the benefit of the KV cache is that you are not holding entire
> > 64K
> > >> blocks in memory when you only care about 200 bytes of them.  Would an
> > >> alternative be to set a small block size (2KB or less)?
> > >>
> > >> The problems with small block sizes would be expensive block cache
> > >> management overhead and inefficient scanning IO due to lack of
> > read-ahead.
> > >>  Maybe improving the cache management and read-ahead would be more
> > general
> > >> improvements that don't add as much complexity?
> > >>
> > >> I'm having a hard time envisioning how you would do invalidations on
> > the KV
> > >> cache and how you would merge its entries into a scan, etc.  Would it
> > >> basically be a memstore in front of the memstore where KVs get
> > individually
> > >> invalidated instead of bulk-flushed?  Would it be sorted or hashed?
> > >>
> > >> Matt
> > >>
> > >> On Wed, Apr 4, 2012 at 10:35 AM, Enis Söztutar <[hidden email]>
> wrote:
> > >>
> > >> > As you said, caching the entire row does not make much sense, given
> > that
> > >> > the families are by contract the access boundaries. But caching
> column
> > >> > families might be a good trade of for dealing with the per-item
> > overhead.
> > >> >
> > >> > Also agreed on cache being configurable at the table or better cf
> > level.
> > >> I
> > >> > think we can do something like enable_block_cache = true,
> > >> > enable_kv_cache=false, per column family.
> > >> >
> > >> > Enis
> > >> >
> > >> > On Tue, Apr 3, 2012 at 11:03 PM, Vladimir Rodionov
> > >> > <[hidden email]>wrote:
> > >> >
> > >> > > Usually make sense for tables with random mostly access (point
> > queries)
> > >> > > For short-long scans block cache is preferable.
> > >> > > Cassandra has it (Row cache) but as since they cache the whole row
> > >> (which
> > >> > > can be very large) in many cases
> > >> > > it has sub par performance. Make sense to make caching
> configurable:
> > >> > table
> > >> > > can use key-value cache and do not use block cache
> > >> > > and vice verse.
> > >> > >
> > >> > > Best regards,
> > >> > > Vladimir Rodionov
> > >> > > Principal Platform Engineer
> > >> > > Carrier IQ, www.carrieriq.com
> > >> > > e-mail: [hidden email]
> > >> > >
> > >> > > ________________________________________
> > >> > > From: Enis Söztutar [[hidden email]]
> > >> > > Sent: Tuesday, April 03, 2012 3:34 PM
> > >> > > To: [hidden email]
> > >> > > Subject: keyvalue cache
> > >> > >
> > >> > > Hi,
> > >> > >
> > >> > > Before opening the issue, I though I should ask around first. What
> > do
> > >> you
> > >> > > think about a keyvalue cache sitting on top of the block cache? It
> > is
> > >> > > mentioned in the big table paper, and it seems that zipfian kv
> > access
> > >> > > patterns might benefit from something like this a lot. I could not
> > find
> > >> > > anybody who proposed that before.
> > >> > >
> > >> > > What do you guys think? Should we pursue a kv query-cache. My gut
> > >> feeling
> > >> > > says that especially for some workloads we might gain significant
> > >> > > performance improvements, but we cannot verify it, until we
> > implement
> > >> and
> > >> > > profile it, right?
> > >> > >
> > >> > > Thanks,
> > >> > > Enis
> > >> > >
> > >> > > Confidentiality Notice:  The information contained in this
> message,
> > >> > > including any attachments hereto, may be confidential and is
> > intended
> > >> to
> > >> > be
> > >> > > read only by the individual or entity to whom this message is
> > >> addressed.
> > >> > If
> > >> > > the reader of this message is not the intended recipient or an
> > agent or
> > >> > > designee of the intended recipient, please note that any review,
> > use,
> > >> > > disclosure or distribution of this message or its attachments, in
> > any
> > >> > form,
> > >> > > is strictly prohibited.  If you have received this message in
> error,
> > >> > please
> > >> > > immediately notify the sender and/or
> [hidden email]
> > >> > > delete or destroy any copy of this message and its attachments.
> > >> > >
> > >> >
> > >>
> > >> Confidentiality Notice:  The information contained in this message,
> > >> including any attachments hereto, may be confidential and is intended
> > to be
> > >> read only by the individual or entity to whom this message is
> > addressed. If
> > >> the reader of this message is not the intended recipient or an agent
> or
> > >> designee of the intended recipient, please note that any review, use,
> > >> disclosure or distribution of this message or its attachments, in any
> > form,
> > >> is strictly prohibited.  If you have received this message in error,
> > please
> > >> immediately notify the sender and/or [hidden email] and
> > >> delete or destroy any copy of this message and its attachments.
> > >>
> > >
> > >
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: keyvalue cache

Matt Corgan
random afterthought: a delete-row command could cause problems for a simple
HashSet style cache, so you would need a sorted implementation or a more
elaborate nested structure.


On Wed, Apr 4, 2012 at 3:55 PM, Enis Söztutar <[hidden email]> wrote:

> I think you are right that if you replicate the row MVCC semantics in the
> cache, then
> you can get a consistent view. I was referring to a more client side
> approach.
>
> I guess the take aways are:
>  - forget about scans, and shoot for point gets, which I agree
>  - per-kv cache overhead might be huge, but still worth trying it out.
>  - can also be architected on top of coprocessors
>  - might be complex to implement, but still some use cases would benefit a
> lot.
>  - row cache / family cache / kv cache
>
> Thanks for all the input!
>
> Enis
>
> On Wed, Apr 4, 2012 at 3:28 PM, Matt Corgan <[hidden email]> wrote:
>
> > A client-side memcached setup can stay pretty consistent if you send all
> of
> > your puts and deletes through it before sending them to hbase, but yeah,
> I
> > guess you lose strict consistency under heavy read/write from multiple
> > simultaneous clients.  But, like Andy is saying, if you route the
> requests
> > through the regionserver and it talks to memcached/hazelcast, couldn't
> that
> > be fully consistent?
> >
> >
> > On Wed, Apr 4, 2012 at 3:09 PM, Andrew Purtell <[hidden email]>
> > wrote:
> >
> > > I thought about trying this out once with a coprocessor, hooking the
> > Gets,
> > > with an embedded Hazelcast. That would just be a proof of concept. The
> > idea
> > > is to scale the KV cache independent of regionserver limits (maybe
> we're
> > > only giving 1 GB per RS to the value cache and a 10 GB region is hot)
> and
> > > the next step could be modifying the client to spread read load over
> > > replicas (HBASE-2357). This doesn't consider scans either.
> > >
> > >
> > > Best regards,
> > >
> > >     - Andy
> > >
> > >
> > > Problems worthy of attack prove their worth by hitting back. - Piet
> Hein
> > > (via Tom White)
> > >
> > >
> > >
> > > >________________________________
> > > > From: Matt Corgan <[hidden email]>
> > > >To: [hidden email]
> > > >Sent: Wednesday, April 4, 2012 2:46 PM
> > > >Subject: Re: keyvalue cache
> > > >
> > > >It could act like a HashSet of KeyValues keyed on the
> > > >rowKey+family+qualifier but not including the timestamp.  As writes
> come
> > > in
> > > >it would evict or overwrite previous versions (read-through vs
> > > >write-through).  It would only service point queries where the
> > > >row+fam+qualifier are specified, returning the latest version.
>  Wouldn't
> > > be
> > > >able to do a typical rowKey-only Get (scan behind the scenes) because
> it
> > > >wouldn't know if it contained all the cells in the row, but if you
> could
> > > >specify all your row's qualifiers up-front it could work.
> > > >
> > > >
> > > >On Wed, Apr 4, 2012 at 2:30 PM, Vladimir Rodionov
> > > ><[hidden email]>wrote:
> > > >
> > > >> 1. 2KB can be too large for some applications. For example, some of
> > our
> > > >> k-v sizes < 100 bytes combined.
> > > >> 2. These tables (from 1.) do not benefit from block cache at all (we
> > did
> > > >> not try 100 B block size yet :)
> > > >> 3. And Matt is absolutely right: small block size is expensive
> > > >>
> > > >> How about doing point queries on K-V cache and  bypass K-V cache on
> > all
> > > >> Scans (when someone really need this)?
> > > >> Implement K-V cache as a coprocessor application?
> > > >>
> > > >> Invalidation of K-V entry is not necessary if all upserts operations
> > go
> > > >> through K-V cache firstly if it sits in front of MemStore.
> > > >> There will be no "stale or invalid" data situation in this case.
> > > Correct?
> > > >> No need for data to be sorted and no need for data to be merged
> > > >> into a scan (we do not use K-V cache for Scans)
> > > >>
> > > >>
> > > >> Best regards,
> > > >> Vladimir Rodionov
> > > >> Principal Platform Engineer
> > > >> Carrier IQ, www.carrieriq.com
> > > >> e-mail: [hidden email]
> > > >>
> > > >> ________________________________________
> > > >> From: Matt Corgan [[hidden email]]
> > > >> Sent: Wednesday, April 04, 2012 11:40 AM
> > > >> To: [hidden email]
> > > >> Subject: Re: keyvalue cache
> > > >>
> > > >> I guess the benefit of the KV cache is that you are not holding
> entire
> > > 64K
> > > >> blocks in memory when you only care about 200 bytes of them.  Would
> an
> > > >> alternative be to set a small block size (2KB or less)?
> > > >>
> > > >> The problems with small block sizes would be expensive block cache
> > > >> management overhead and inefficient scanning IO due to lack of
> > > read-ahead.
> > > >>  Maybe improving the cache management and read-ahead would be more
> > > general
> > > >> improvements that don't add as much complexity?
> > > >>
> > > >> I'm having a hard time envisioning how you would do invalidations on
> > > the KV
> > > >> cache and how you would merge its entries into a scan, etc.  Would
> it
> > > >> basically be a memstore in front of the memstore where KVs get
> > > individually
> > > >> invalidated instead of bulk-flushed?  Would it be sorted or hashed?
> > > >>
> > > >> Matt
> > > >>
> > > >> On Wed, Apr 4, 2012 at 10:35 AM, Enis Söztutar <[hidden email]>
> > wrote:
> > > >>
> > > >> > As you said, caching the entire row does not make much sense,
> given
> > > that
> > > >> > the families are by contract the access boundaries. But caching
> > column
> > > >> > families might be a good trade of for dealing with the per-item
> > > overhead.
> > > >> >
> > > >> > Also agreed on cache being configurable at the table or better cf
> > > level.
> > > >> I
> > > >> > think we can do something like enable_block_cache = true,
> > > >> > enable_kv_cache=false, per column family.
> > > >> >
> > > >> > Enis
> > > >> >
> > > >> > On Tue, Apr 3, 2012 at 11:03 PM, Vladimir Rodionov
> > > >> > <[hidden email]>wrote:
> > > >> >
> > > >> > > Usually make sense for tables with random mostly access (point
> > > queries)
> > > >> > > For short-long scans block cache is preferable.
> > > >> > > Cassandra has it (Row cache) but as since they cache the whole
> row
> > > >> (which
> > > >> > > can be very large) in many cases
> > > >> > > it has sub par performance. Make sense to make caching
> > configurable:
> > > >> > table
> > > >> > > can use key-value cache and do not use block cache
> > > >> > > and vice verse.
> > > >> > >
> > > >> > > Best regards,
> > > >> > > Vladimir Rodionov
> > > >> > > Principal Platform Engineer
> > > >> > > Carrier IQ, www.carrieriq.com
> > > >> > > e-mail: [hidden email]
> > > >> > >
> > > >> > > ________________________________________
> > > >> > > From: Enis Söztutar [[hidden email]]
> > > >> > > Sent: Tuesday, April 03, 2012 3:34 PM
> > > >> > > To: [hidden email]
> > > >> > > Subject: keyvalue cache
> > > >> > >
> > > >> > > Hi,
> > > >> > >
> > > >> > > Before opening the issue, I though I should ask around first.
> What
> > > do
> > > >> you
> > > >> > > think about a keyvalue cache sitting on top of the block cache?
> It
> > > is
> > > >> > > mentioned in the big table paper, and it seems that zipfian kv
> > > access
> > > >> > > patterns might benefit from something like this a lot. I could
> not
> > > find
> > > >> > > anybody who proposed that before.
> > > >> > >
> > > >> > > What do you guys think? Should we pursue a kv query-cache. My
> gut
> > > >> feeling
> > > >> > > says that especially for some workloads we might gain
> significant
> > > >> > > performance improvements, but we cannot verify it, until we
> > > implement
> > > >> and
> > > >> > > profile it, right?
> > > >> > >
> > > >> > > Thanks,
> > > >> > > Enis
> > > >> > >
> > > >> > > Confidentiality Notice:  The information contained in this
> > message,
> > > >> > > including any attachments hereto, may be confidential and is
> > > intended
> > > >> to
> > > >> > be
> > > >> > > read only by the individual or entity to whom this message is
> > > >> addressed.
> > > >> > If
> > > >> > > the reader of this message is not the intended recipient or an
> > > agent or
> > > >> > > designee of the intended recipient, please note that any review,
> > > use,
> > > >> > > disclosure or distribution of this message or its attachments,
> in
> > > any
> > > >> > form,
> > > >> > > is strictly prohibited.  If you have received this message in
> > error,
> > > >> > please
> > > >> > > immediately notify the sender and/or
> > [hidden email]
> > > >> > > delete or destroy any copy of this message and its attachments.
> > > >> > >
> > > >> >
> > > >>
> > > >> Confidentiality Notice:  The information contained in this message,
> > > >> including any attachments hereto, may be confidential and is
> intended
> > > to be
> > > >> read only by the individual or entity to whom this message is
> > > addressed. If
> > > >> the reader of this message is not the intended recipient or an agent
> > or
> > > >> designee of the intended recipient, please note that any review,
> use,
> > > >> disclosure or distribution of this message or its attachments, in
> any
> > > form,
> > > >> is strictly prohibited.  If you have received this message in error,
> > > please
> > > >> immediately notify the sender and/or [hidden email]
> > > >> delete or destroy any copy of this message and its attachments.
> > > >>
> > > >
> > > >
> > > >
> > >
> >
>
Loading...