Showing posts with label decimal. Show all posts
Showing posts with label decimal. Show all posts

Friday, March 9, 2012

Matrix - Problem with Rounding in Subtotal Column

Hi,

I have a matrix report that calculates data and provides a subtotal at the end. The calculated value is displayed to one decimal place.

My problem is that when the subtotal appears, it appears to one decimal place also, and although it is the correct value, it is not actually a sum of the values above it.

Here's what I mean:

Client Formatted Data Unformatted Data A 1.6 1.578947368 B 0.8 0.789473684 C 0.8 0.789473684 D -0.8 -0.789473684 E 1.6 1.578947368 Total 3.9 3.947368421

The formatted data row is what I have on my matrix report - the Users are looking at this and complaining that the Total row = 3.9, but the sum of the values above it is 4.0. If you look at the raw data I have included (this is not shown on the report), 3.9 is actually the correct value, but you can understand where they are coming from when they can't see this.

It seems that the Subtotal is aggregated before any formatting applies, so its not actually a subtotal of the visible data in the cells in its group, but a subtotal of the raw data in the cells in its group.

Does anyone know how to solve this?

Thanks !

hi

if u used any formating for the fields like format(a,0),format(b,0) the same you need to use when u calculate total like format(a,0)+format(b,0) etc .i hope if u use this you will get the correct total.

|||

I think your problem is that you're doing the formatting after Aggregation.. for example you're probably using something like Format(Sum(<FieldName>)). Your problem in that case will be resolved if you do it before Aggregation that is, something like Sum(Format(<FieldName>))

-Aayush

|||If I use Sum(Format(<FieldName>)) I get an error saying that "the expression uses an aggregate function on data that is not numeric", even if I format it as a number.|||

That is because the Format functions convert data to String type. So you can probably use a conversion function after Formatting, that is something like

Sum(CDbl(Format(<FieldName>))). I know its a very crude kind of fix but I guess it doesn't matter as long as it works. There might be a better way but at least I'm not aware of one.

-Aayush

|||This worked, thanks !

Matrix - Problem with Rounding in Subtotal Column

Hi,

I have a matrix report that calculates data and provides a subtotal at the end. The calculated value is displayed to one decimal place.

My problem is that when the subtotal appears, it appears to one decimal place also, and although it is the correct value, it is not actually a sum of the values above it.

Here's what I mean:

Client Formatted Data Unformatted Data A 1.6 1.578947368 B 0.8 0.789473684 C 0.8 0.789473684 D -0.8 -0.789473684 E 1.6 1.578947368 Total 3.9 3.947368421

The formatted data row is what I have on my matrix report - the Users are looking at this and complaining that the Total row = 3.9, but the sum of the values above it is 4.0. If you look at the raw data I have included (this is not shown on the report), 3.9 is actually the correct value, but you can understand where they are coming from when they can't see this.

It seems that the Subtotal is aggregated before any formatting applies, so its not actually a subtotal of the visible data in the cells in its group, but a subtotal of the raw data in the cells in its group.

Does anyone know how to solve this?

Thanks !

hi

if u used any formating for the fields like format(a,0),format(b,0) the same you need to use when u calculate total like format(a,0)+format(b,0) etc .i hope if u use this you will get the correct total.

|||

I think your problem is that you're doing the formatting after Aggregation.. for example you're probably using something like Format(Sum(<FieldName>)). Your problem in that case will be resolved if you do it before Aggregation that is, something like Sum(Format(<FieldName>))

-Aayush

|||If I use Sum(Format(<FieldName>)) I get an error saying that "the expression uses an aggregate function on data that is not numeric", even if I format it as a number.|||

That is because the Format functions convert data to String type. So you can probably use a conversion function after Formatting, that is something like

Sum(CDbl(Format(<FieldName>))). I know its a very crude kind of fix but I guess it doesn't matter as long as it works. There might be a better way but at least I'm not aware of one.

-Aayush

|||This worked, thanks !

Wednesday, March 7, 2012

math error

select convert(float,'1.2334e+006')
1233400.0

select convert(decimal(20,2),'1.2334e+006')
Server: Msg 8114, Level 16, State 5, Line 1
Error converting data type varchar to numeric.

can I set some options arithabort etc to have a workaround to this
problem?
Thanks.othellomy@.yahoo.com wrote:

Quote:

Originally Posted by

select convert(float,'1.2334e+006')
1233400.0
>
select convert(decimal(20,2),'1.2334e+006')
Server: Msg 8114, Level 16, State 5, Line 1
Error converting data type varchar to numeric.
>
can I set some options arithabort etc to have a workaround to this
problem?
Thanks.


Try

select convert(decimal(20,2),1.2334e+006)

Madhivanan|||Hi Madhivanan,
Thanks for the input. The column type is varchar where the value
1.2334e+006 is stored. So I need to convert it to float as suggested by
some before converting it to decimal. Anyway, that brings another
issue. When running the query the server just errors out without giving
the value that causes the error. It just says 'conversion error'. I
had to take the SQl out and put it in a cursor and use a loop to find
out which row is actually causing the error and find the value
1.2334e+006. Is there any easier way to find out which row in the table
causes the SQL server to error out. For example can I set the error
level so that I find more information so that I can locate the row in
the table.
Thanks.

Madhivanan wrote:

Quote:

Originally Posted by

othellomy@.yahoo.com wrote:

Quote:

Originally Posted by

select convert(float,'1.2334e+006')
1233400.0

select convert(decimal(20,2),'1.2334e+006')
Server: Msg 8114, Level 16, State 5, Line 1
Error converting data type varchar to numeric.

can I set some options arithabort etc to have a workaround to this
problem?
Thanks.


>
Try
>
select convert(decimal(20,2),1.2334e+006)
>
Madhivanan

|||(othellomy@.yahoo.com) writes:

Quote:

Originally Posted by

Hi Madhivanan,
Thanks for the input. The column type is varchar where the value
1.2334e+006 is stored. So I need to convert it to float as suggested by
some before converting it to decimal. Anyway, that brings another
issue. When running the query the server just errors out without giving
the value that causes the error. It just says 'conversion error'. I
had to take the SQl out and put it in a cursor and use a loop to find
out which row is actually causing the error and find the value
1.2334e+006. Is there any easier way to find out which row in the table
causes the SQL server to error out. For example can I set the error
level so that I find more information so that I can locate the row in
the table.


Unfortunately, there are not really any good options. The best is probably
to run a SELECT query, and takes some hint from where it terminates. This
example illustrates:

CREATE TABLE #tmp1 (a varchar(23) NOT NULL, b int IDENTITY)
go
INSERT #tmp1(a) VALUES ('1234')
INSERT #tmp1(a) VALUES ('1232')
INSERT #tmp1(a) VALUES ('2344')
INSERT #tmp1(a) VALUES ('34.34')
INSERT #tmp1(a) VALUES ('-1234')
INSERT #tmp1(a) VALUES ('-1234')
INSERT #tmp1(a) VALUES ('1234e+006')
INSERT #tmp1(a) VALUES ('777')
go
SELECT convert(decimal(20, 2), a) FROM #tmp1 ORDER BY b
go
DROP TABLE #tmp1

--
Erland Sommarskog, SQL Server MVP, esquel@.sommarskog.se
Books Online for SQL Server 2005 at
http://www.microsoft.com/technet/pr...oads/books.mspx
Books Online for SQL Server 2000 at
http://www.microsoft.com/sql/prodin...ions/books.mspx