[DISCUSS] hbase-2.0.0 compatibility expectations

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[DISCUSS] hbase-2.0.0 compatibility expectations

stack-3
What are our expectations regards compatibility between hbase1 and hbase2?

Lets have a chat about it. Here are some goal posts.

+ You have to upgrade to hbase-1.x before you can migrate to hbase-2. No
migration from < hbase-1 (Is this too onerous? Should we support 0.98 =>
2.0?).
+ You do NOT have to upgrade to the latest release of hbase1 to migrate to
hbase2; being up on hbase-1.0.0+ will be sufficient.
+ You'll have to update your hbase1 coprocessors to deploy them on hbase2.
A bunch of CP API has/will change by the time hbase2 comes out; e.g.
watching for region split on RegionServer no longer makes sense given
Master runs all splits now.
+ An hbase1 client can run against an hbase2 cluster but it will only be
able to do DML (Get/Put/Scan, etc.). We do not allow being able to do admin
ops using an hbase1 Admin client against an hbase2 cluster. We have some
egregious API violations in branch-1; e.g. we have protobuf in our API (See
HBASE-15607). The notion is that we can't afford a deprecation cycle
purging this stuff from our Admin API.

What you all think?

St.Ack
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [DISCUSS] hbase-2.0.0 compatibility expectations

Zach York-2
Do we know what the major pain points for migration are? Can we discuss
that/get a list going?

I think without that knowledge it is hard (for me at least :) ) to
determine where we should set our sights in terms of migration.

Thanks,
Zach

On Wed, Aug 2, 2017 at 4:38 PM, Stack <[hidden email]> wrote:

> What are our expectations regards compatibility between hbase1 and hbase2?
>
> Lets have a chat about it. Here are some goal posts.
>
> + You have to upgrade to hbase-1.x before you can migrate to hbase-2. No
> migration from < hbase-1 (Is this too onerous? Should we support 0.98 =>
> 2.0?).
> + You do NOT have to upgrade to the latest release of hbase1 to migrate to
> hbase2; being up on hbase-1.0.0+ will be sufficient.
> + You'll have to update your hbase1 coprocessors to deploy them on hbase2.
> A bunch of CP API has/will change by the time hbase2 comes out; e.g.
> watching for region split on RegionServer no longer makes sense given
> Master runs all splits now.
> + An hbase1 client can run against an hbase2 cluster but it will only be
> able to do DML (Get/Put/Scan, etc.). We do not allow being able to do admin
> ops using an hbase1 Admin client against an hbase2 cluster. We have some
> egregious API violations in branch-1; e.g. we have protobuf in our API (See
> HBASE-15607). The notion is that we can't afford a deprecation cycle
> purging this stuff from our Admin API.
>
> What you all think?
>
> St.Ack
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [DISCUSS] hbase-2.0.0 compatibility expectations

stack-3
On Wed, Aug 2, 2017 at 5:25 PM, Zach York <[hidden email]>
wrote:

> Do we know what the major pain points for migration are? Can we discuss
> that/get a list going?
>
>
Here's a few in outline:

+ There is issue of formats, of hbase-2.x being able to read hbase-1.x data
whether from HDFS or ZooKeeper or off the wire.
+ An hbase-1.x client should be able to Get/Put and Scan an hbase-2.x
cluster; no holes in the API or unintelligible serializations.
+ There is then the little dance that has us rolling restart from an
hbase-1.x cluster to hbase-2.x; i.e. upgrade master first and then it will
assign regions to the new hbase-2.x regionservers as they come on line. TBD.

Is this what you mean sir?

S


> I think without that knowledge it is hard (for me at least :) ) to
> determine where we should set our sights in terms of migration.
>
> Thanks,
> Zach
>
> On Wed, Aug 2, 2017 at 4:38 PM, Stack <[hidden email]> wrote:
>
> > What are our expectations regards compatibility between hbase1 and
> hbase2?
> >
> > Lets have a chat about it. Here are some goal posts.
> >
> > + You have to upgrade to hbase-1.x before you can migrate to hbase-2. No
> > migration from < hbase-1 (Is this too onerous? Should we support 0.98 =>
> > 2.0?).
> > + You do NOT have to upgrade to the latest release of hbase1 to migrate
> to
> > hbase2; being up on hbase-1.0.0+ will be sufficient.
> > + You'll have to update your hbase1 coprocessors to deploy them on
> hbase2.
> > A bunch of CP API has/will change by the time hbase2 comes out; e.g.
> > watching for region split on RegionServer no longer makes sense given
> > Master runs all splits now.
> > + An hbase1 client can run against an hbase2 cluster but it will only be
> > able to do DML (Get/Put/Scan, etc.). We do not allow being able to do
> admin
> > ops using an hbase1 Admin client against an hbase2 cluster. We have some
> > egregious API violations in branch-1; e.g. we have protobuf in our API
> (See
> > HBASE-15607). The notion is that we can't afford a deprecation cycle
> > purging this stuff from our Admin API.
> >
> > What you all think?
> >
> > St.Ack
> >
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [DISCUSS] hbase-2.0.0 compatibility expectations

Zach York-2
This kinda helps, but these seem more like expectations. I was going more
for things like HFile format changed, meta table structure changed,
coprocessor implementations changed (these are just examples, I don't know
if any of these actually changed).

More technical differences between branch-1 and branch-2 which then can
help us get the right expectations for compatibility.

On Wed, Aug 2, 2017 at 6:34 PM, Stack <[hidden email]> wrote:

> On Wed, Aug 2, 2017 at 5:25 PM, Zach York <[hidden email]>
> wrote:
>
> > Do we know what the major pain points for migration are? Can we discuss
> > that/get a list going?
> >
> >
> Here's a few in outline:
>
> + There is issue of formats, of hbase-2.x being able to read hbase-1.x data
> whether from HDFS or ZooKeeper or off the wire.
> + An hbase-1.x client should be able to Get/Put and Scan an hbase-2.x
> cluster; no holes in the API or unintelligible serializations.
> + There is then the little dance that has us rolling restart from an
> hbase-1.x cluster to hbase-2.x; i.e. upgrade master first and then it will
> assign regions to the new hbase-2.x regionservers as they come on line.
> TBD.
>
> Is this what you mean sir?
>
> S
>
>
> > I think without that knowledge it is hard (for me at least :) ) to
> > determine where we should set our sights in terms of migration.
> >
> > Thanks,
> > Zach
> >
> > On Wed, Aug 2, 2017 at 4:38 PM, Stack <[hidden email]> wrote:
> >
> > > What are our expectations regards compatibility between hbase1 and
> > hbase2?
> > >
> > > Lets have a chat about it. Here are some goal posts.
> > >
> > > + You have to upgrade to hbase-1.x before you can migrate to hbase-2.
> No
> > > migration from < hbase-1 (Is this too onerous? Should we support 0.98
> =>
> > > 2.0?).
> > > + You do NOT have to upgrade to the latest release of hbase1 to migrate
> > to
> > > hbase2; being up on hbase-1.0.0+ will be sufficient.
> > > + You'll have to update your hbase1 coprocessors to deploy them on
> > hbase2.
> > > A bunch of CP API has/will change by the time hbase2 comes out; e.g.
> > > watching for region split on RegionServer no longer makes sense given
> > > Master runs all splits now.
> > > + An hbase1 client can run against an hbase2 cluster but it will only
> be
> > > able to do DML (Get/Put/Scan, etc.). We do not allow being able to do
> > admin
> > > ops using an hbase1 Admin client against an hbase2 cluster. We have
> some
> > > egregious API violations in branch-1; e.g. we have protobuf in our API
> > (See
> > > HBASE-15607). The notion is that we can't afford a deprecation cycle
> > > purging this stuff from our Admin API.
> > >
> > > What you all think?
> > >
> > > St.Ack
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [DISCUSS] hbase-2.0.0 compatibility expectations

Zach York-2
I tried to filter based on imcompatible labels and there were only two
JIRAs returned [1]. I have a hard time believing that there were only two
breaking changes from 1.x to 2.x.

[1]
https://issues.apache.org/jira/browse/HBASE-17957?jql=project%20%3D%20HBASE%20AND%20fixVersion%20%3D%202.0.0%20AND%20labels%20in%20(incompatibleChange%2C%20incompatible%2C%20incompatibility)

On Thu, Aug 3, 2017 at 8:54 AM, Zach York <[hidden email]>
wrote:

> This kinda helps, but these seem more like expectations. I was going more
> for things like HFile format changed, meta table structure changed,
> coprocessor implementations changed (these are just examples, I don't know
> if any of these actually changed).
>
> More technical differences between branch-1 and branch-2 which then can
> help us get the right expectations for compatibility.
>
> On Wed, Aug 2, 2017 at 6:34 PM, Stack <[hidden email]> wrote:
>
>> On Wed, Aug 2, 2017 at 5:25 PM, Zach York <[hidden email]>
>> wrote:
>>
>> > Do we know what the major pain points for migration are? Can we discuss
>> > that/get a list going?
>> >
>> >
>> Here's a few in outline:
>>
>> + There is issue of formats, of hbase-2.x being able to read hbase-1.x
>> data
>> whether from HDFS or ZooKeeper or off the wire.
>> + An hbase-1.x client should be able to Get/Put and Scan an hbase-2.x
>> cluster; no holes in the API or unintelligible serializations.
>> + There is then the little dance that has us rolling restart from an
>> hbase-1.x cluster to hbase-2.x; i.e. upgrade master first and then it will
>> assign regions to the new hbase-2.x regionservers as they come on line.
>> TBD.
>>
>> Is this what you mean sir?
>>
>> S
>>
>>
>> > I think without that knowledge it is hard (for me at least :) ) to
>> > determine where we should set our sights in terms of migration.
>> >
>> > Thanks,
>> > Zach
>> >
>> > On Wed, Aug 2, 2017 at 4:38 PM, Stack <[hidden email]> wrote:
>> >
>> > > What are our expectations regards compatibility between hbase1 and
>> > hbase2?
>> > >
>> > > Lets have a chat about it. Here are some goal posts.
>> > >
>> > > + You have to upgrade to hbase-1.x before you can migrate to hbase-2.
>> No
>> > > migration from < hbase-1 (Is this too onerous? Should we support 0.98
>> =>
>> > > 2.0?).
>> > > + You do NOT have to upgrade to the latest release of hbase1 to
>> migrate
>> > to
>> > > hbase2; being up on hbase-1.0.0+ will be sufficient.
>> > > + You'll have to update your hbase1 coprocessors to deploy them on
>> > hbase2.
>> > > A bunch of CP API has/will change by the time hbase2 comes out; e.g.
>> > > watching for region split on RegionServer no longer makes sense given
>> > > Master runs all splits now.
>> > > + An hbase1 client can run against an hbase2 cluster but it will only
>> be
>> > > able to do DML (Get/Put/Scan, etc.). We do not allow being able to do
>> > admin
>> > > ops using an hbase1 Admin client against an hbase2 cluster. We have
>> some
>> > > egregious API violations in branch-1; e.g. we have protobuf in our API
>> > (See
>> > > HBASE-15607). The notion is that we can't afford a deprecation cycle
>> > > purging this stuff from our Admin API.
>> > >
>> > > What you all think?
>> > >
>> > > St.Ack
>> > >
>> >
>>
>
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [DISCUSS] hbase-2.0.0 compatibility expectations

Ted Yu-3
I expanded the condition in the filter like this:

project = HBASE AND fixVersion in (2.0.0, 2.0.0-alpha-1,  2.0.0-alpha-2,
3.0) AND labels in (incompatibleChange, incompatible, incompatibility)

Still there're only two showing up.

On Thu, Aug 3, 2017 at 9:07 AM, Zach York <[hidden email]>
wrote:

> I tried to filter based on imcompatible labels and there were only two
> JIRAs returned [1]. I have a hard time believing that there were only two
> breaking changes from 1.x to 2.x.
>
> [1]
> https://issues.apache.org/jira/browse/HBASE-17957?jql=
> project%20%3D%20HBASE%20AND%20fixVersion%20%3D%202.0.0%
> 20AND%20labels%20in%20(incompatibleChange%2C%20incompatible%2C%
> 20incompatibility)
>
> On Thu, Aug 3, 2017 at 8:54 AM, Zach York <[hidden email]>
> wrote:
>
> > This kinda helps, but these seem more like expectations. I was going more
> > for things like HFile format changed, meta table structure changed,
> > coprocessor implementations changed (these are just examples, I don't
> know
> > if any of these actually changed).
> >
> > More technical differences between branch-1 and branch-2 which then can
> > help us get the right expectations for compatibility.
> >
> > On Wed, Aug 2, 2017 at 6:34 PM, Stack <[hidden email]> wrote:
> >
> >> On Wed, Aug 2, 2017 at 5:25 PM, Zach York <[hidden email]
> >
> >> wrote:
> >>
> >> > Do we know what the major pain points for migration are? Can we
> discuss
> >> > that/get a list going?
> >> >
> >> >
> >> Here's a few in outline:
> >>
> >> + There is issue of formats, of hbase-2.x being able to read hbase-1.x
> >> data
> >> whether from HDFS or ZooKeeper or off the wire.
> >> + An hbase-1.x client should be able to Get/Put and Scan an hbase-2.x
> >> cluster; no holes in the API or unintelligible serializations.
> >> + There is then the little dance that has us rolling restart from an
> >> hbase-1.x cluster to hbase-2.x; i.e. upgrade master first and then it
> will
> >> assign regions to the new hbase-2.x regionservers as they come on line.
> >> TBD.
> >>
> >> Is this what you mean sir?
> >>
> >> S
> >>
> >>
> >> > I think without that knowledge it is hard (for me at least :) ) to
> >> > determine where we should set our sights in terms of migration.
> >> >
> >> > Thanks,
> >> > Zach
> >> >
> >> > On Wed, Aug 2, 2017 at 4:38 PM, Stack <[hidden email]> wrote:
> >> >
> >> > > What are our expectations regards compatibility between hbase1 and
> >> > hbase2?
> >> > >
> >> > > Lets have a chat about it. Here are some goal posts.
> >> > >
> >> > > + You have to upgrade to hbase-1.x before you can migrate to
> hbase-2.
> >> No
> >> > > migration from < hbase-1 (Is this too onerous? Should we support
> 0.98
> >> =>
> >> > > 2.0?).
> >> > > + You do NOT have to upgrade to the latest release of hbase1 to
> >> migrate
> >> > to
> >> > > hbase2; being up on hbase-1.0.0+ will be sufficient.
> >> > > + You'll have to update your hbase1 coprocessors to deploy them on
> >> > hbase2.
> >> > > A bunch of CP API has/will change by the time hbase2 comes out; e.g.
> >> > > watching for region split on RegionServer no longer makes sense
> given
> >> > > Master runs all splits now.
> >> > > + An hbase1 client can run against an hbase2 cluster but it will
> only
> >> be
> >> > > able to do DML (Get/Put/Scan, etc.). We do not allow being able to
> do
> >> > admin
> >> > > ops using an hbase1 Admin client against an hbase2 cluster. We have
> >> some
> >> > > egregious API violations in branch-1; e.g. we have protobuf in our
> API
> >> > (See
> >> > > HBASE-15607). The notion is that we can't afford a deprecation cycle
> >> > > purging this stuff from our Admin API.
> >> > >
> >> > > What you all think?
> >> > >
> >> > > St.Ack
> >> > >
> >> >
> >>
> >
> >
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [DISCUSS] hbase-2.0.0 compatibility expectations

stack-5
In reply to this post by Zach York-2
Thanks Zach for clarification. Let me work up a list and then come back to
this thread.  Jira needs an edit pass to.

S

On Aug 3, 2017 23:54, "Zach York" <[hidden email]> wrote:

This kinda helps, but these seem more like expectations. I was going more
for things like HFile format changed, meta table structure changed,
coprocessor implementations changed (these are just examples, I don't know
if any of these actually changed).

More technical differences between branch-1 and branch-2 which then can
help us get the right expectations for compatibility.

On Wed, Aug 2, 2017 at 6:34 PM, Stack <[hidden email]> wrote:

> On Wed, Aug 2, 2017 at 5:25 PM, Zach York <[hidden email]>
> wrote:
>
> > Do we know what the major pain points for migration are? Can we discuss
> > that/get a list going?
> >
> >
> Here's a few in outline:
>
> + There is issue of formats, of hbase-2.x being able to read hbase-1.x
data

> whether from HDFS or ZooKeeper or off the wire.
> + An hbase-1.x client should be able to Get/Put and Scan an hbase-2.x
> cluster; no holes in the API or unintelligible serializations.
> + There is then the little dance that has us rolling restart from an
> hbase-1.x cluster to hbase-2.x; i.e. upgrade master first and then it will
> assign regions to the new hbase-2.x regionservers as they come on line.
> TBD.
>
> Is this what you mean sir?
>
> S
>
>
> > I think without that knowledge it is hard (for me at least :) ) to
> > determine where we should set our sights in terms of migration.
> >
> > Thanks,
> > Zach
> >
> > On Wed, Aug 2, 2017 at 4:38 PM, Stack <[hidden email]> wrote:
> >
> > > What are our expectations regards compatibility between hbase1 and
> > hbase2?
> > >
> > > Lets have a chat about it. Here are some goal posts.
> > >
> > > + You have to upgrade to hbase-1.x before you can migrate to hbase-2.
> No
> > > migration from < hbase-1 (Is this too onerous? Should we support 0.98
> =>
> > > 2.0?).
> > > + You do NOT have to upgrade to the latest release of hbase1 to
migrate

> > to
> > > hbase2; being up on hbase-1.0.0+ will be sufficient.
> > > + You'll have to update your hbase1 coprocessors to deploy them on
> > hbase2.
> > > A bunch of CP API has/will change by the time hbase2 comes out; e.g.
> > > watching for region split on RegionServer no longer makes sense given
> > > Master runs all splits now.
> > > + An hbase1 client can run against an hbase2 cluster but it will only
> be
> > > able to do DML (Get/Put/Scan, etc.). We do not allow being able to do
> > admin
> > > ops using an hbase1 Admin client against an hbase2 cluster. We have
> some
> > > egregious API violations in branch-1; e.g. we have protobuf in our API
> > (See
> > > HBASE-15607). The notion is that we can't afford a deprecation cycle
> > > purging this stuff from our Admin API.
> > >
> > > What you all think?
> > >
> > > St.Ack
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [DISCUSS] hbase-2.0.0 compatibility expectations

Esteban Gutierrez
Should we add additional details around replication as well? for instance,
shall we consider a hbase-1.x cluster as a client for a hbase-2.x cluster?

Thanks for starting this discussion Stack,

esteban.

--
Cloudera, Inc.


On Fri, Aug 4, 2017 at 1:05 AM, stack <[hidden email]> wrote:

> Thanks Zach for clarification. Let me work up a list and then come back to
> this thread.  Jira needs an edit pass to.
>
> S
>
> On Aug 3, 2017 23:54, "Zach York" <[hidden email]> wrote:
>
> This kinda helps, but these seem more like expectations. I was going more
> for things like HFile format changed, meta table structure changed,
> coprocessor implementations changed (these are just examples, I don't know
> if any of these actually changed).
>
> More technical differences between branch-1 and branch-2 which then can
> help us get the right expectations for compatibility.
>
> On Wed, Aug 2, 2017 at 6:34 PM, Stack <[hidden email]> wrote:
>
> > On Wed, Aug 2, 2017 at 5:25 PM, Zach York <[hidden email]>
> > wrote:
> >
> > > Do we know what the major pain points for migration are? Can we discuss
> > > that/get a list going?
> > >
> > >
> > Here's a few in outline:
> >
> > + There is issue of formats, of hbase-2.x being able to read hbase-1.x
> data
> > whether from HDFS or ZooKeeper or off the wire.
> > + An hbase-1.x client should be able to Get/Put and Scan an hbase-2.x
> > cluster; no holes in the API or unintelligible serializations.
> > + There is then the little dance that has us rolling restart from an
> > hbase-1.x cluster to hbase-2.x; i.e. upgrade master first and then it
> will
> > assign regions to the new hbase-2.x regionservers as they come on line.
> > TBD.
> >
> > Is this what you mean sir?
> >
> > S
> >
> >
> > > I think without that knowledge it is hard (for me at least :) ) to
> > > determine where we should set our sights in terms of migration.
> > >
> > > Thanks,
> > > Zach
> > >
> > > On Wed, Aug 2, 2017 at 4:38 PM, Stack <[hidden email]> wrote:
> > >
> > > > What are our expectations regards compatibility between hbase1 and
> > > hbase2?
> > > >
> > > > Lets have a chat about it. Here are some goal posts.
> > > >
> > > > + You have to upgrade to hbase-1.x before you can migrate to hbase-2.
> > No
> > > > migration from < hbase-1 (Is this too onerous? Should we support 0.98
> > =>
> > > > 2.0?).
> > > > + You do NOT have to upgrade to the latest release of hbase1 to
> migrate
> > > to
> > > > hbase2; being up on hbase-1.0.0+ will be sufficient.
> > > > + You'll have to update your hbase1 coprocessors to deploy them on
> > > hbase2.
> > > > A bunch of CP API has/will change by the time hbase2 comes out; e.g.
> > > > watching for region split on RegionServer no longer makes sense given
> > > > Master runs all splits now.
> > > > + An hbase1 client can run against an hbase2 cluster but it will only
> > be
> > > > able to do DML (Get/Put/Scan, etc.). We do not allow being able to do
> > > admin
> > > > ops using an hbase1 Admin client against an hbase2 cluster. We have
> > some
> > > > egregious API violations in branch-1; e.g. we have protobuf in our
> API
> > > (See
> > > > HBASE-15607). The notion is that we can't afford a deprecation cycle
> > > > purging this stuff from our Admin API.
> > > >
> > > > What you all think?
> > > >
> > > > St.Ack
> > > >
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [DISCUSS] hbase-2.0.0 compatibility expectations

Andrew Purtell-3
This is a really good question. I think many operators will give a lot of leeway to data format changes as long as data can be copied from A to B (perhaps with batch rewrite to upgrade (ideally, not required)) and replication can be enabled to sync up to the current moment for cut over.


> On Aug 4, 2017, at 10:00 AM, Esteban Gutierrez <[hidden email]> wrote:
>
> Should we add additional details around replication as well? for instance,
> shall we consider a hbase-1.x cluster as a client for a hbase-2.x cluster?
>
> Thanks for starting this discussion Stack,
>
> esteban.
>
> --
> Cloudera, Inc.
>
>
>> On Fri, Aug 4, 2017 at 1:05 AM, stack <[hidden email]> wrote:
>>
>> Thanks Zach for clarification. Let me work up a list and then come back to
>> this thread.  Jira needs an edit pass to.
>>
>> S
>>
>> On Aug 3, 2017 23:54, "Zach York" <[hidden email]> wrote:
>>
>> This kinda helps, but these seem more like expectations. I was going more
>> for things like HFile format changed, meta table structure changed,
>> coprocessor implementations changed (these are just examples, I don't know
>> if any of these actually changed).
>>
>> More technical differences between branch-1 and branch-2 which then can
>> help us get the right expectations for compatibility.
>>
>>> On Wed, Aug 2, 2017 at 6:34 PM, Stack <[hidden email]> wrote:
>>>
>>> On Wed, Aug 2, 2017 at 5:25 PM, Zach York <[hidden email]>
>>> wrote:
>>>
>>>> Do we know what the major pain points for migration are? Can we discuss
>>>> that/get a list going?
>>>>
>>>>
>>> Here's a few in outline:
>>>
>>> + There is issue of formats, of hbase-2.x being able to read hbase-1.x
>> data
>>> whether from HDFS or ZooKeeper or off the wire.
>>> + An hbase-1.x client should be able to Get/Put and Scan an hbase-2.x
>>> cluster; no holes in the API or unintelligible serializations.
>>> + There is then the little dance that has us rolling restart from an
>>> hbase-1.x cluster to hbase-2.x; i.e. upgrade master first and then it
>> will
>>> assign regions to the new hbase-2.x regionservers as they come on line.
>>> TBD.
>>>
>>> Is this what you mean sir?
>>>
>>> S
>>>
>>>
>>>> I think without that knowledge it is hard (for me at least :) ) to
>>>> determine where we should set our sights in terms of migration.
>>>>
>>>> Thanks,
>>>> Zach
>>>>
>>>>> On Wed, Aug 2, 2017 at 4:38 PM, Stack <[hidden email]> wrote:
>>>>>
>>>>> What are our expectations regards compatibility between hbase1 and
>>>> hbase2?
>>>>>
>>>>> Lets have a chat about it. Here are some goal posts.
>>>>>
>>>>> + You have to upgrade to hbase-1.x before you can migrate to hbase-2.
>>> No
>>>>> migration from < hbase-1 (Is this too onerous? Should we support 0.98
>>> =>
>>>>> 2.0?).
>>>>> + You do NOT have to upgrade to the latest release of hbase1 to
>> migrate
>>>> to
>>>>> hbase2; being up on hbase-1.0.0+ will be sufficient.
>>>>> + You'll have to update your hbase1 coprocessors to deploy them on
>>>> hbase2.
>>>>> A bunch of CP API has/will change by the time hbase2 comes out; e.g.
>>>>> watching for region split on RegionServer no longer makes sense given
>>>>> Master runs all splits now.
>>>>> + An hbase1 client can run against an hbase2 cluster but it will only
>>> be
>>>>> able to do DML (Get/Put/Scan, etc.). We do not allow being able to do
>>>> admin
>>>>> ops using an hbase1 Admin client against an hbase2 cluster. We have
>>> some
>>>>> egregious API violations in branch-1; e.g. we have protobuf in our
>> API
>>>> (See
>>>>> HBASE-15607). The notion is that we can't afford a deprecation cycle
>>>>> purging this stuff from our Admin API.
>>>>>
>>>>> What you all think?
>>>>>
>>>>> St.Ack
>>>>>
>>>>
>>>
>>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [DISCUSS] hbase-2.0.0 compatibility expectations

Apekshit Sharma
Ref: deleting/deprecating some methods in HBaseAdmin
https://github.com/apache/hbase/commit/de696cf6b653749c6bf105ef3d62d7a6c6923c57
From first message:
*" An hbase1 client can run against an hbase2 cluster but it will only be
able to do DML (Get/Put/Scan, etc.). We do not allow being able to do admin
ops using an hbase1 Admin client against an hbase2 cluster."*

Question 1:
These guarantees are with what version of jars?
Does the client has to keep using corresponding 1.x jars with which it was
compiled, OR can the 2.x jars just replace earlier ones and client is still
expected to work correctly? (i would say no!)

Question second:
How do we handle clients doing deprecated admin ops?
a) make admin functions no-op by using empty method.
b) throw exception. May/maynot crash the client - ideally shouldn't since
methods are define with 'throws IOException'.
c) delete the method - will crash the client
But the question only makes sense if expect clients to work with 2.x jars.
If not, we can just delete them, simple!

Question 3: AMv2 related
There are ops in old HBaseAdmin that'll be destructive for 2.0
(closeRegion() fn).
If clients continue to use 1.x jars and make these calls, that's not
acceptable. We need to handle such deprecation on server sider itself.
If only there was way of versioning the rpc service!
Another way, but bad one is, deprecate existing ones and move logic to a
new one. I think there are just 1-2 that need this treatment.
Either way, need to come up with a solution!


On Fri, Aug 4, 2017 at 10:08 AM, Andrew Purtell <[hidden email]>
wrote:

> This is a really good question. I think many operators will give a lot of
> leeway to data format changes as long as data can be copied from A to B
> (perhaps with batch rewrite to upgrade (ideally, not required)) and
> replication can be enabled to sync up to the current moment for cut over.
>
>
> > On Aug 4, 2017, at 10:00 AM, Esteban Gutierrez <[hidden email]>
> wrote:
> >
> > Should we add additional details around replication as well? for
> instance,
> > shall we consider a hbase-1.x cluster as a client for a hbase-2.x
> cluster?
> >
> > Thanks for starting this discussion Stack,
> >
> > esteban.
> >
> > --
> > Cloudera, Inc.
> >
> >
> >> On Fri, Aug 4, 2017 at 1:05 AM, stack <[hidden email]> wrote:
> >>
> >> Thanks Zach for clarification. Let me work up a list and then come back
> to
> >> this thread.  Jira needs an edit pass to.
> >>
> >> S
> >>
> >> On Aug 3, 2017 23:54, "Zach York" <[hidden email]> wrote:
> >>
> >> This kinda helps, but these seem more like expectations. I was going
> more
> >> for things like HFile format changed, meta table structure changed,
> >> coprocessor implementations changed (these are just examples, I don't
> know
> >> if any of these actually changed).
> >>
> >> More technical differences between branch-1 and branch-2 which then can
> >> help us get the right expectations for compatibility.
> >>
> >>> On Wed, Aug 2, 2017 at 6:34 PM, Stack <[hidden email]> wrote:
> >>>
> >>> On Wed, Aug 2, 2017 at 5:25 PM, Zach York <
> [hidden email]>
> >>> wrote:
> >>>
> >>>> Do we know what the major pain points for migration are? Can we
> discuss
> >>>> that/get a list going?
> >>>>
> >>>>
> >>> Here's a few in outline:
> >>>
> >>> + There is issue of formats, of hbase-2.x being able to read hbase-1.x
> >> data
> >>> whether from HDFS or ZooKeeper or off the wire.
> >>> + An hbase-1.x client should be able to Get/Put and Scan an hbase-2.x
> >>> cluster; no holes in the API or unintelligible serializations.
> >>> + There is then the little dance that has us rolling restart from an
> >>> hbase-1.x cluster to hbase-2.x; i.e. upgrade master first and then it
> >> will
> >>> assign regions to the new hbase-2.x regionservers as they come on line.
> >>> TBD.
> >>>
> >>> Is this what you mean sir?
> >>>
> >>> S
> >>>
> >>>
> >>>> I think without that knowledge it is hard (for me at least :) ) to
> >>>> determine where we should set our sights in terms of migration.
> >>>>
> >>>> Thanks,
> >>>> Zach
> >>>>
> >>>>> On Wed, Aug 2, 2017 at 4:38 PM, Stack <[hidden email]> wrote:
> >>>>>
> >>>>> What are our expectations regards compatibility between hbase1 and
> >>>> hbase2?
> >>>>>
> >>>>> Lets have a chat about it. Here are some goal posts.
> >>>>>
> >>>>> + You have to upgrade to hbase-1.x before you can migrate to hbase-2.
> >>> No
> >>>>> migration from < hbase-1 (Is this too onerous? Should we support 0.98
> >>> =>
> >>>>> 2.0?).
> >>>>> + You do NOT have to upgrade to the latest release of hbase1 to
> >> migrate
> >>>> to
> >>>>> hbase2; being up on hbase-1.0.0+ will be sufficient.
> >>>>> + You'll have to update your hbase1 coprocessors to deploy them on
> >>>> hbase2.
> >>>>> A bunch of CP API has/will change by the time hbase2 comes out; e.g.
> >>>>> watching for region split on RegionServer no longer makes sense given
> >>>>> Master runs all splits now.
> >>>>> + An hbase1 client can run against an hbase2 cluster but it will only
> >>> be
> >>>>> able to do DML (Get/Put/Scan, etc.). We do not allow being able to do
> >>>> admin
> >>>>> ops using an hbase1 Admin client against an hbase2 cluster. We have
> >>> some
> >>>>> egregious API violations in branch-1; e.g. we have protobuf in our
> >> API
> >>>> (See
> >>>>> HBASE-15607). The notion is that we can't afford a deprecation cycle
> >>>>> purging this stuff from our Admin API.
> >>>>>
> >>>>> What you all think?
> >>>>>
> >>>>> St.Ack
> >>>>>
> >>>>
> >>>
> >>
>



--

-- Appy
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [DISCUSS] hbase-2.0.0 compatibility expectations

stack-3
In reply to this post by Esteban Gutierrez
On Fri, Aug 4, 2017 at 10:00 AM, Esteban Gutierrez <[hidden email]>
wrote:

> Should we add additional details around replication as well? for instance,
> shall we consider a hbase-1.x cluster as a client for a hbase-2.x cluster?
>
>
Yes. I'd say this should be a blocker Esteban. Filed HBASE-18596.
Thanks,
S





> Thanks for starting this discussion Stack,
>
> esteban.
>
> --
> Cloudera, Inc.
>
>
> On Fri, Aug 4, 2017 at 1:05 AM, stack <[hidden email]> wrote:
>
> > Thanks Zach for clarification. Let me work up a list and then come back
> to
> > this thread.  Jira needs an edit pass to.
> >
> > S
> >
> > On Aug 3, 2017 23:54, "Zach York" <[hidden email]> wrote:
> >
> > This kinda helps, but these seem more like expectations. I was going more
> > for things like HFile format changed, meta table structure changed,
> > coprocessor implementations changed (these are just examples, I don't
> know
> > if any of these actually changed).
> >
> > More technical differences between branch-1 and branch-2 which then can
> > help us get the right expectations for compatibility.
> >
> > On Wed, Aug 2, 2017 at 6:34 PM, Stack <[hidden email]> wrote:
> >
> > > On Wed, Aug 2, 2017 at 5:25 PM, Zach York <
> [hidden email]>
> > > wrote:
> > >
> > > > Do we know what the major pain points for migration are? Can we
> discuss
> > > > that/get a list going?
> > > >
> > > >
> > > Here's a few in outline:
> > >
> > > + There is issue of formats, of hbase-2.x being able to read hbase-1.x
> > data
> > > whether from HDFS or ZooKeeper or off the wire.
> > > + An hbase-1.x client should be able to Get/Put and Scan an hbase-2.x
> > > cluster; no holes in the API or unintelligible serializations.
> > > + There is then the little dance that has us rolling restart from an
> > > hbase-1.x cluster to hbase-2.x; i.e. upgrade master first and then it
> > will
> > > assign regions to the new hbase-2.x regionservers as they come on line.
> > > TBD.
> > >
> > > Is this what you mean sir?
> > >
> > > S
> > >
> > >
> > > > I think without that knowledge it is hard (for me at least :) ) to
> > > > determine where we should set our sights in terms of migration.
> > > >
> > > > Thanks,
> > > > Zach
> > > >
> > > > On Wed, Aug 2, 2017 at 4:38 PM, Stack <[hidden email]> wrote:
> > > >
> > > > > What are our expectations regards compatibility between hbase1 and
> > > > hbase2?
> > > > >
> > > > > Lets have a chat about it. Here are some goal posts.
> > > > >
> > > > > + You have to upgrade to hbase-1.x before you can migrate to
> hbase-2.
> > > No
> > > > > migration from < hbase-1 (Is this too onerous? Should we support
> 0.98
> > > =>
> > > > > 2.0?).
> > > > > + You do NOT have to upgrade to the latest release of hbase1 to
> > migrate
> > > > to
> > > > > hbase2; being up on hbase-1.0.0+ will be sufficient.
> > > > > + You'll have to update your hbase1 coprocessors to deploy them on
> > > > hbase2.
> > > > > A bunch of CP API has/will change by the time hbase2 comes out;
> e.g.
> > > > > watching for region split on RegionServer no longer makes sense
> given
> > > > > Master runs all splits now.
> > > > > + An hbase1 client can run against an hbase2 cluster but it will
> only
> > > be
> > > > > able to do DML (Get/Put/Scan, etc.). We do not allow being able to
> do
> > > > admin
> > > > > ops using an hbase1 Admin client against an hbase2 cluster. We have
> > > some
> > > > > egregious API violations in branch-1; e.g. we have protobuf in our
> > API
> > > > (See
> > > > > HBASE-15607). The notion is that we can't afford a deprecation
> cycle
> > > > > purging this stuff from our Admin API.
> > > > >
> > > > > What you all think?
> > > > >
> > > > > St.Ack
> > > > >
> > > >
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [DISCUSS] hbase-2.0.0 compatibility expectations

stack-3
In reply to this post by Apekshit Sharma
On Fri, Aug 11, 2017 at 5:07 PM, Apekshit Sharma <[hidden email]> wrote:

> Ref: deleting/deprecating some methods in HBaseAdmin
> https://github.com/apache/hbase/commit/de696cf6b653749c6bf105ef3d62d7
> a6c6923c57
> From first message:
> *" An hbase1 client can run against an hbase2 cluster but it will only be
> able to do DML (Get/Put/Scan, etc.). We do not allow being able to do admin
> ops using an hbase1 Admin client against an hbase2 cluster."*
>
> Question 1:
> These guarantees are with what version of jars?
> Does the client has to keep using corresponding 1.x jars with which it was
> compiled, OR can the 2.x jars just replace earlier ones and client is still
> expected to work correctly? (i would say no!)
>
>
Dropping hbase2 jars into an hbase1 deploy? No. No guarantee of binary
compat.

I was thinking of wire compatibility here Appy.



> Question second:
> How do we handle clients doing deprecated admin ops?
> a) make admin functions no-op by using empty method.
> b) throw exception. May/maynot crash the client - ideally shouldn't since
> methods are define with 'throws IOException'.
>

I think throwing exception the way to go. Its clean signal of change.


> c) delete the method - will crash the client
> But the question only makes sense if expect clients to work with 2.x jars.
> If not, we can just delete them, simple!
>
>
I'm not sure I follow. If a hbase1 client calls a deprecated/unimplemented
method, I was thinking it'd get an exception.



> Question 3: AMv2 related
> There are ops in old HBaseAdmin that'll be destructive for 2.0
> (closeRegion() fn).
> If clients continue to use 1.x jars and make these calls, that's not
> acceptable. We need to handle such deprecation on server sider itself.
> If only there was way of versioning the rpc service!
> Another way, but bad one is, deprecate existing ones and move logic to a
> new one. I think there are just 1-2 that need this treatment.
> Either way, need to come up with a solution!
>
>
Throw an exception pointing at the alternative API?

Thanks Appy,
S


>
> On Fri, Aug 4, 2017 at 10:08 AM, Andrew Purtell <[hidden email]>
> wrote:
>
> > This is a really good question. I think many operators will give a lot of
> > leeway to data format changes as long as data can be copied from A to B
> > (perhaps with batch rewrite to upgrade (ideally, not required)) and
> > replication can be enabled to sync up to the current moment for cut over.
> >
> >
> > > On Aug 4, 2017, at 10:00 AM, Esteban Gutierrez <[hidden email]>
> > wrote:
> > >
> > > Should we add additional details around replication as well? for
> > instance,
> > > shall we consider a hbase-1.x cluster as a client for a hbase-2.x
> > cluster?
> > >
> > > Thanks for starting this discussion Stack,
> > >
> > > esteban.
> > >
> > > --
> > > Cloudera, Inc.
> > >
> > >
> > >> On Fri, Aug 4, 2017 at 1:05 AM, stack <[hidden email]> wrote:
> > >>
> > >> Thanks Zach for clarification. Let me work up a list and then come
> back
> > to
> > >> this thread.  Jira needs an edit pass to.
> > >>
> > >> S
> > >>
> > >> On Aug 3, 2017 23:54, "Zach York" <[hidden email]>
> wrote:
> > >>
> > >> This kinda helps, but these seem more like expectations. I was going
> > more
> > >> for things like HFile format changed, meta table structure changed,
> > >> coprocessor implementations changed (these are just examples, I don't
> > know
> > >> if any of these actually changed).
> > >>
> > >> More technical differences between branch-1 and branch-2 which then
> can
> > >> help us get the right expectations for compatibility.
> > >>
> > >>> On Wed, Aug 2, 2017 at 6:34 PM, Stack <[hidden email]> wrote:
> > >>>
> > >>> On Wed, Aug 2, 2017 at 5:25 PM, Zach York <
> > [hidden email]>
> > >>> wrote:
> > >>>
> > >>>> Do we know what the major pain points for migration are? Can we
> > discuss
> > >>>> that/get a list going?
> > >>>>
> > >>>>
> > >>> Here's a few in outline:
> > >>>
> > >>> + There is issue of formats, of hbase-2.x being able to read
> hbase-1.x
> > >> data
> > >>> whether from HDFS or ZooKeeper or off the wire.
> > >>> + An hbase-1.x client should be able to Get/Put and Scan an hbase-2.x
> > >>> cluster; no holes in the API or unintelligible serializations.
> > >>> + There is then the little dance that has us rolling restart from an
> > >>> hbase-1.x cluster to hbase-2.x; i.e. upgrade master first and then it
> > >> will
> > >>> assign regions to the new hbase-2.x regionservers as they come on
> line.
> > >>> TBD.
> > >>>
> > >>> Is this what you mean sir?
> > >>>
> > >>> S
> > >>>
> > >>>
> > >>>> I think without that knowledge it is hard (for me at least :) ) to
> > >>>> determine where we should set our sights in terms of migration.
> > >>>>
> > >>>> Thanks,
> > >>>> Zach
> > >>>>
> > >>>>> On Wed, Aug 2, 2017 at 4:38 PM, Stack <[hidden email]> wrote:
> > >>>>>
> > >>>>> What are our expectations regards compatibility between hbase1 and
> > >>>> hbase2?
> > >>>>>
> > >>>>> Lets have a chat about it. Here are some goal posts.
> > >>>>>
> > >>>>> + You have to upgrade to hbase-1.x before you can migrate to
> hbase-2.
> > >>> No
> > >>>>> migration from < hbase-1 (Is this too onerous? Should we support
> 0.98
> > >>> =>
> > >>>>> 2.0?).
> > >>>>> + You do NOT have to upgrade to the latest release of hbase1 to
> > >> migrate
> > >>>> to
> > >>>>> hbase2; being up on hbase-1.0.0+ will be sufficient.
> > >>>>> + You'll have to update your hbase1 coprocessors to deploy them on
> > >>>> hbase2.
> > >>>>> A bunch of CP API has/will change by the time hbase2 comes out;
> e.g.
> > >>>>> watching for region split on RegionServer no longer makes sense
> given
> > >>>>> Master runs all splits now.
> > >>>>> + An hbase1 client can run against an hbase2 cluster but it will
> only
> > >>> be
> > >>>>> able to do DML (Get/Put/Scan, etc.). We do not allow being able to
> do
> > >>>> admin
> > >>>>> ops using an hbase1 Admin client against an hbase2 cluster. We have
> > >>> some
> > >>>>> egregious API violations in branch-1; e.g. we have protobuf in our
> > >> API
> > >>>> (See
> > >>>>> HBASE-15607). The notion is that we can't afford a deprecation
> cycle
> > >>>>> purging this stuff from our Admin API.
> > >>>>>
> > >>>>> What you all think?
> > >>>>>
> > >>>>> St.Ack
> > >>>>>
> > >>>>
> > >>>
> > >>
> >
>
>
>
> --
>
> -- Appy
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [DISCUSS] hbase-2.0.0 compatibility expectations

Esteban Gutierrez
In reply to this post by stack-3
Thanks Stack!

--
Cloudera, Inc.


On Mon, Aug 14, 2017 at 1:05 PM, Stack <[hidden email]> wrote:

> On Fri, Aug 4, 2017 at 10:00 AM, Esteban Gutierrez <[hidden email]>
> wrote:
>
> > Should we add additional details around replication as well? for
> instance,
> > shall we consider a hbase-1.x cluster as a client for a hbase-2.x
> cluster?
> >
> >
> Yes. I'd say this should be a blocker Esteban. Filed HBASE-18596.
> Thanks,
> S
>
>
>
>
>
> > Thanks for starting this discussion Stack,
> >
> > esteban.
> >
> > --
> > Cloudera, Inc.
> >
> >
> > On Fri, Aug 4, 2017 at 1:05 AM, stack <[hidden email]> wrote:
> >
> > > Thanks Zach for clarification. Let me work up a list and then come back
> > to
> > > this thread.  Jira needs an edit pass to.
> > >
> > > S
> > >
> > > On Aug 3, 2017 23:54, "Zach York" <[hidden email]>
> wrote:
> > >
> > > This kinda helps, but these seem more like expectations. I was going
> more
> > > for things like HFile format changed, meta table structure changed,
> > > coprocessor implementations changed (these are just examples, I don't
> > know
> > > if any of these actually changed).
> > >
> > > More technical differences between branch-1 and branch-2 which then can
> > > help us get the right expectations for compatibility.
> > >
> > > On Wed, Aug 2, 2017 at 6:34 PM, Stack <[hidden email]> wrote:
> > >
> > > > On Wed, Aug 2, 2017 at 5:25 PM, Zach York <
> > [hidden email]>
> > > > wrote:
> > > >
> > > > > Do we know what the major pain points for migration are? Can we
> > discuss
> > > > > that/get a list going?
> > > > >
> > > > >
> > > > Here's a few in outline:
> > > >
> > > > + There is issue of formats, of hbase-2.x being able to read
> hbase-1.x
> > > data
> > > > whether from HDFS or ZooKeeper or off the wire.
> > > > + An hbase-1.x client should be able to Get/Put and Scan an hbase-2.x
> > > > cluster; no holes in the API or unintelligible serializations.
> > > > + There is then the little dance that has us rolling restart from an
> > > > hbase-1.x cluster to hbase-2.x; i.e. upgrade master first and then it
> > > will
> > > > assign regions to the new hbase-2.x regionservers as they come on
> line.
> > > > TBD.
> > > >
> > > > Is this what you mean sir?
> > > >
> > > > S
> > > >
> > > >
> > > > > I think without that knowledge it is hard (for me at least :) ) to
> > > > > determine where we should set our sights in terms of migration.
> > > > >
> > > > > Thanks,
> > > > > Zach
> > > > >
> > > > > On Wed, Aug 2, 2017 at 4:38 PM, Stack <[hidden email]> wrote:
> > > > >
> > > > > > What are our expectations regards compatibility between hbase1
> and
> > > > > hbase2?
> > > > > >
> > > > > > Lets have a chat about it. Here are some goal posts.
> > > > > >
> > > > > > + You have to upgrade to hbase-1.x before you can migrate to
> > hbase-2.
> > > > No
> > > > > > migration from < hbase-1 (Is this too onerous? Should we support
> > 0.98
> > > > =>
> > > > > > 2.0?).
> > > > > > + You do NOT have to upgrade to the latest release of hbase1 to
> > > migrate
> > > > > to
> > > > > > hbase2; being up on hbase-1.0.0+ will be sufficient.
> > > > > > + You'll have to update your hbase1 coprocessors to deploy them
> on
> > > > > hbase2.
> > > > > > A bunch of CP API has/will change by the time hbase2 comes out;
> > e.g.
> > > > > > watching for region split on RegionServer no longer makes sense
> > given
> > > > > > Master runs all splits now.
> > > > > > + An hbase1 client can run against an hbase2 cluster but it will
> > only
> > > > be
> > > > > > able to do DML (Get/Put/Scan, etc.). We do not allow being able
> to
> > do
> > > > > admin
> > > > > > ops using an hbase1 Admin client against an hbase2 cluster. We
> have
> > > > some
> > > > > > egregious API violations in branch-1; e.g. we have protobuf in
> our
> > > API
> > > > > (See
> > > > > > HBASE-15607). The notion is that we can't afford a deprecation
> > cycle
> > > > > > purging this stuff from our Admin API.
> > > > > >
> > > > > > What you all think?
> > > > > >
> > > > > > St.Ack
> > > > > >
> > > > >
> > > >
> > >
> >
>
Loading...